Research Articles



Commentary: Augmented Reality Scientific Phone Apps –making the APD AR Holistic Review app and using existing AR apps for scientific publications


Authors: Jun-Jie Poh 1,3, Ser-Xian Phua 1, Kwok-Fong Chan 1,Samuel Ken-En Gan 1,2,3,#


1 Bioinformatics Institute, Agency for Science, Technology and Research (A*STAR),Singapore 138671
2 p53 Laboratory, Agency for Science, Technology and Research (A*STAR),Singapore
3 APD SKEG Pte Ltd, Singapore 529757

Received: 26 September 2018 | Accepted: 28 September 2018 | Published: 2 October 2018

Scientific Phone Apps and Mobile Devices (2018) 4:4

DOI:10.30943/2018/28092018

INTRODUCTION

The average scientific publication is not the most palatable of reading materials, especially to those not in the relevant disciplines. Yet, conveying scientific concepts easily is precisely what scientific publications are meant to do. Imagine what the use of video pictures/paintings as depicted in the Harry Potter movie series can do to make things easier!

While the fantasy of moving photos/pictures does not exist physically in the real world, just as Santa Claus cannot travel the world today without being shot down by anti-missiles, there are some shadows of reality in this idea. Apart from the popular Instagram app where short video clips can be used as video pictures, other technology in the form of Augmented Reality (AR) can make this “magic” a reality in communicating challenging scientific concepts in an interesting and attractive manner.

Utilized in popular game apps like “Pokemon Go”, AR can also be used in scientific publications to illustrate biological structures and concepts. With animated AR, the typical scientific paper can masquerade as published research from Harry Potter’s Hogwarts School of Witchcraft and Wizardry.

While the use of AR had been gaining traction, mostly towards video games, education and entertainment, AR has yet to penetrate fully into scientific publications where it can play an important role to address the difficulty of squeezing three-dimensional (3D) ideas into the traditional two-dimensional (2D) graphics on screen/paper.


AR APPLICATION TO SCIENTIFIC PUBLISHING

Scientific publishing is the bread and butter of academic research for the sciences, and the onus is on the authors to convey their work to the scientific community and the general public (Gan, 2018a, 2018b). Beyond academic research, it is also the responsibility of the educators and presenters to have their materials easily accessible and comprehensible. This often involves the use of graphics embedded in slide presentations in lectures. In chemistry and biology, structures are difficult to convey in even the best descriptions. Even pictures, which is said to be more than “a thousand words”, are limited. This is especially so given the emergence of complex protein crystal structures that had augmented the application of drug design and discovery (Carvalho et al., 2010; Chiang et al., 2018; Kang et al., 2018).

In the example of protein models, visualization had traditionally relied stereo-images (McConathy and Owens, 2003). Today, we have software such as PyMOL (Schrödinger, LLC, 2015) that is widely used to visualize 3D structures. Nonetheless, stereoscopic views of protein structures remain the best option on printed material and on non-interactive platforms on screens (e.g. webpage, PDF, etc), but also requiring trained viewers. These stereo-images (see examples in Ban, 2000; Henderson et al., 1990), are limited in their 3D information, requiring trained crossing of eyes, and are now rarely used. This thus opens the possibility of AR.

AR apps have the potential to be companion scientific apps to scientific articles and are likely to be the next revolution of digital scientific publishing. In addition, these are likely to be a common type of scientific phone apps of the future. While it would be ridiculous to have every scientific publication having its own mobile AR app to burden the reader, we wish to illustrate here the potential use of such AR apps in simplifying the communication of complex scientific concepts and ideas.

Coupled with animation, AR apps can better communicate concepts and ideas, providing an additional dimension without changing the graphic figures, past (previously published), present, and future

To demonstrate how AR apps can be incorporated into the process of scientific communication, we take the example of our previously published paper (Figure 2 from Su et al. 2017) and a perspective review paper that we have submitted titled “An Augmented Reality illustrated perspective for a holistic view of proteins: impact on antibody engineering, and drug discovery". Both articles incorporated many structural biology illustrations. To demonstrate how the proteins interact with each other, we incorporated AR animations to show protein binding and movement. This was performed using Unity and MAXST.AR software to create the “APD AR Holistic Review” apps and also to utilize existing AR app – HP Revealed (formerly known as Aurasma).


Figure 1. Screenshot of APD AR Holistic Review App with AR depiction of immunoglobulin G antibody. The publication graphic is hidden under the AR circular plate.


Development and Usage of APD AR Holistic Review

We used UCSF Chimera 1.11.2 (Pettersen et al., 2004) to generate static 3-D protein models from PDB file format trajectories before exporting the X3D file format for Blender 2.79which was used to convert the X3D file to FBX file for theUnity3Dengine.

Unity version 2017.3 was used to further animate the 3D models to project a four-dimensional view of the protein with movement Using the MAXST.AR software development kit ver. 3.5, an Image Tracker GameObject was created, with a 2dmap file format generated from MAXST Developer website using the original graphic from the publication attached to the Image Tracker

The 3D animated model is then attached to the Image Tracker which would then be projected whenever when the smartphone camera detects the publication graphic. This process was built using the Unity software. Both iOS and Android apps were build using the Unity software before uploading to the respective app stores

To use the APD AR Holistic review app, simple download and install the application, activate it, and point the smartphone camera at the publication graphic. The 3D animated AR projection of the figure will appear. The camera can be moved to view around the animation so long the image remains detected.

The app is available in Google Play Store and Apple App Store.


Development and Usage of HP Reveal AR models

As was performed above for the APD AR Holistic App Review app with the difference that the X3D file was converted to a DAE file. Decimation (reduction of triangles in a model) was performed on the 3D protein structures to fit approximately 15000 triangles for an optimum performance as suggested in HP Reveal 3D guidelines.

The publication graphic was uploaded as a “Trigger Image” into HP Reveal Studio Aura Creation. The thumbnail, texture image files and protein DAE file were compressed and uploaded as an “Overlay”. The 3D protein model is linked to the publication graphic as an Aura and can be viewed on the application as soon as the model has been saved on the Studio platform.

To use the application, simple install and access the HP Revealed app that will open the HP Reveal scanner. Simply point the smartphone camera at the publication graphic and the AR model will appear. Move the camera or the image target to view the 3D protein structure from more perspectives.

For private sharing of the Auras, access “My Account” in HP Reveal Studio. Users may Share Campaign to obtain a link for sharing the AR models for other mobile devices that has the installed application.

HP Reveal is available in Google Play Store and Apple App Store.

While the review we have written is currently under review. The title could be searched for in time for viewing the AR, but in the meantime, Figure 2 from Su et al., 2017 may be used for viewing the AR.


Figure 2. Augmented reality depiction of HIV-1 gag polyprotein from Figure 2 of Su et al. (2017) displayed on the HP Reveal App.


Main Considerations

One of the disadvantages of the APD AR Holistic Review is the size of the application package. Currently, the 3D models and the image tracker library are embedded in the application, which makes the app file sizeable. The future may include uploading models onto cloud storage which would then be fetched when the image tracker detects the publication graphic, reducing significantly the download file size of the application. Nonetheless, this may incur delays as the files would then be downloaded via the Internet live. This then is a balance between a one time large file download of the app or an ad-hoc download of the animations. With the inevitable increase in computational power in the future, an AR of a full molecular dynamics simulation using its trajectories may be possible.

In the utilization of the HP Reveal App, we had to reduce the protein model quality by flattening the surface model in order to allow a reasonable experience using the HP Reveal App.

The accuracy of the protein surface representation is thus lowered. In addition, the text orientation is fixed to the model, which makes it difficult to read at certain angles.

In the development of our own AR app, we had issues in listing the app in the Apple App Store which initially found the app to contain “minimal functionality”. In overcoming this, we had to add instructions and other features for it to be approved.

Our work is a proof in concept to the possibility of incorporating more AR technology as scientific apps into scientific publishing in which is exciting due to its ability to convey graphic information better. Using the example from Figure 2 of Su et al., 2017, structural proteins crammed into a two-dimensional graphics are often intimidating and technical for readers not familiar with the field. With AR, concepts and ideas can be visualized by the reader with minimal technical knowledge.

In addition, we believe the effort to better complemented with animation that would also convey key time-based events revealed in molecular simulations of proteins.


Future Work

There is still great room to grow for scientific phone apps in scientific and medical applications (as previously reviewed in Gan, 2018c; Gan and Poon, 2016) where they can replace laboratory equipment and make scientific research more convenient (in examples as shown in Ng et al., 2017; Sim et al., 2015; Wong et al., 2016), AR can be incorporated into these equipment replacing apps in time. With the development of mobile phone and cloud computing technical capabilities, tapping into more computational power to generate coarse grained and atomistic simulations in the future would be possible.

Contact: samuelg@bii.a-star.edu.sg


Acknowledgements

We would like to thank Dr. Chinh Tran-To Su for providing the structural trajectories of HIV-gag protein and Kenneth Yong Yu Wei for helping in the publishing of the iOS version of APD AR Holistic Review App.


AUTHOR CONTRIBUTIONS

JJP, SXP, KFC and SKEG prepared the manuscript. KFC and SXP prepared the 3D models. JJP developed the mobile apps. KFC designed the HP Reveal AR models. SKEG conceived the idea and supervised all aspects of the manuscript.


References

Ban, N., 2000. The Complete Atomic Structure of the Large Ribosomal Subunit at 2.4 A Resolution. Science 289, 905–920. https://doi.org/10.1126/science.289.5481.905

Carvalho, A.L., Trincão, J., Romão, M.J., 2010. X-Ray Crystallography in Drug Discovery, in: Roque, A.C.A. (Ed.), Ligand-Macromolecular Interactions in Drug Discovery. Humana Press, Totowa, NJ, pp. 31–56. https://doi.org/10.1007/978-1-60761-244-5_3

Chiang, R.Z.-H., Gan, S.K.-E., Su, C.T.-T., 2018. A computational study for rational HIV-1 non-nucleoside reverse transcriptase inhibitor selection and the discovery of novel allosteric pockets for inhibitor design. Bioscience Reports 38, BSR20171113.https://doi.org/10.1042/BSR20171113

Gan, S.K.-E., 2018a. Editorial: APD Trove Journal – scope, aims, and purpose. APD Trove Journal 1. https://doi.org/10.30943/2018/312018

Gan, S.K.-E., 2018b. Editorial: A New Home for “Scientific Phone Apps and Mobile Devices” Journal. Scientific Phone Apps and Mobile Devices 4, 1–3. https://doi.org/10.30943/2018/26042018

Gan, S.K.-E., 2018c. The history and future of scientific phone apps and mobile devices. Scientific Phone Apps and Mobile Devices 4. https://doi.org/10.1186/s41070-018-0022-8

Gan, S.K.-E., Poon, J.-K., 2016. The world of biomedical apps: their uses, limitations, and potential. Scientific Phone Apps and Mobile Devices 2. https://doi.org/10.1186/s41070-016-0009-2

Henderson, R., Baldwin, J.M., Ceska, T.A., Zemlin, F., Beckmann, E., Downing, K.H., 1990. Model for the structure of bacteriorhodopsin based on high-resolution electron cryo-microscopy. Journal of Molecular Biology 213, 899–929. https://doi.org/10.1016/S0022-2836(05)80271-2

Kang, D., Wang, Z., Zhang, H., Wu, G., Zhao, T., Zhou, Z., Huo, Z., Huang, B., Feng, D., Ding, X., Zhang, J., Zuo, X., Jing, L., Luo, W., Guma, S., Daelemans, D., Clercq, E.D., Pannecouque, C., Zhan, P., Liu, X., 2018. Further Exploring Solvent-Exposed Tolerant Regions of Allosteric Binding Pocket for Novel HIV-1 NNRTIs Discovery. ACS Medicinal Chemistry Letters 9, 370–375. https://doi.org/10.1021/acsmedchemlett.8b00054

McConathy, J., Owens, M.J., 2003. Stereochemistry in Drug Action. The Primary Care Companion to The Journal of Clinical Psychiatry 05, 70–73. https://doi.org/10.4088/PCC.v05n0202

Ng, K.M., Wong, C.-F., Liang, A.X., Liew, Y.-H., Yeo, J.Y., Lua, W.-H., Qian, X.-J., Gan, S.K.-E., 2017. APD SpectBT: Ardiono-based mobile vis-Spectrophotometer. Nat Methods Application Notes.

Pettersen, E.F., Goddard, T.D., Huang, C.C., Couch, G.S., Greenblatt, D.M., Meng, E.C., Ferrin, T.E., 2004. UCSF Chimera?A visualization system for exploratory research and analysis. Journal of Computational Chemistry 25, 1605–1612. https://doi.org/10.1002/jcc.20084

Schrödinger, LLC, 2015. The PyMOL Molecular Graphics System, Version 1.8.

Sim, J.-Z., Nguyen, P.-V., Lee, H.-K., Gan, S.K.-E., 2015. GelApp: Mobile gel electrophoresis analyzer. Nat Methods Application Notes.

Su, C.T.-T., Kwoh, C.-K., Verma, C.S., Gan, S.K.-E., 2017. Modeling the full length HIV-1 Gag polyprotein reveals the role of its p6 subunit in viral maturation and the effect of non-cleavage site mutations in protease drug resistance. Journal of Biomolecular Structure and Dynamics 1–12. https://doi.org/10.1080/07391102.2017.1417160

Ban, N., 2000. The Complete Atomic Structure of the Large Ribosomal Subunit at 2.4 A Resolution. Science 289, 905–920. https://doi.org/10.1126/science.289.5481.905

Carvalho, A.L., Trincão, J., Romão, M.J., 2010. X-Ray Crystallography in Drug Discovery, in: Roque, A.C.A. (Ed.), Ligand-Macromolecular Interactions in Drug Discovery. Humana Press, Totowa, NJ, pp. 31–56. https://doi.org/10.1007/978-1-60761-244-5_3

Chiang, R.Z.-H., Gan, S.K.-E., Su, C.T.-T., 2018. A computational study for rational HIV-1 non-nucleoside reverse transcriptase inhibitor selection and the discovery of novel allosteric pockets for inhibitor design. Bioscience Reports 38, BSR20171113. https://doi.org/10.1042/BSR20171113

Gan, S.K.-E., 2018a. Editorial: APD Trove Journal – scope, aims, and purpose. APD Trove Journal 1. https://doi.org/10.30943/2018/312018

Gan, S.K.-E., 2018b. Editorial: A New Home for “Scientific Phone Apps and Mobile Devices” Journal. Scientific Phone Apps and Mobile Devices 4, 1–3. https://doi.org/10.30943/2018/26042018

Gan, S.K.-E., 2018c. The history and future of scientific phone apps and mobile devices. Scientific Phone Apps and Mobile Devices 4. https://doi.org/10.1186/s41070-018-0022-8

Gan, S.K.-E., Poon, J.-K., 2016. The world of biomedical apps: their uses, limitations, and potential. Scientific Phone Apps and Mobile Devices 2. https://doi.org/10.1186/s41070-016-0009-2

Henderson, R., Baldwin, J.M., Ceska, T.A., Zemlin, F., Beckmann, E., Downing, K.H., 1990. Model for the structure of bacteriorhodopsin based on high-resolution electron cryo-microscopy. Journal of Molecular Biology 213, 899–929. https://doi.org/10.1016/S0022-2836(05)80271-2

Kang, D., Wang, Z., Zhang, H., Wu, G., Zhao, T., Zhou, Z., Huo, Z., Huang, B., Feng, D., Ding, X., Zhang, J., Zuo, X., Jing, L., Luo, W., Guma, S., Daelemans, D., Clercq, E.D., Pannecouque, C., Zhan, P., Liu, X., 2018. Further Exploring Solvent-Exposed Tolerant Regions of Allosteric Binding Pocket for Novel HIV-1 NNRTIs Discovery. ACS Medicinal Chemistry Letters 9, 370–375. https://doi.org/10.1021/acsmedchemlett.8b00054

McConathy, J., Owens, M.J., 2003. Stereochemistry in Drug Action. The Primary Care Companion to The Journal of Clinical Psychiatry 05, 70–73. https://doi.org/10.4088/PCC.v05n0202

Ng, K.M., Wong, C.-F., Liang, A.X., Liew, Y.-H., Yeo, J.Y., Lua, W.-H., Qian, X.-J., Gan, S.K.-E., 2017. APD SpectBT: Ardiono-based mobile vis-Spectrophotometer. Nat Methods Application Notes.

Pettersen, E.F., Goddard, T.D., Huang, C.C., Couch, G.S., Greenblatt, D.M., Meng, E.C., Ferrin, T.E., 2004. UCSF Chimera?A visualization system for exploratory research and analysis. Journal of Computational Chemistry 25, 1605–1612. https://doi.org/10.1002/jcc.20084

Schrödinger, LLC, 2015. The PyMOL Molecular Graphics System, Version 1.8.

Sim, J.-Z., Nguyen, P.-V., Lee, H.-K., Gan, S.K.-E., 2015. GelApp: Mobile gel electrophoresis analyzer. Nat Methods Application Notes.

Su, C.T.-T., Kwoh, C.-K., Verma, C.S., Gan, S.K.-E., 2017. Modeling the full length HIV-1 Gag polyprotein reveals the role of its p6 subunit in viral maturation and the effect of non-cleavage site mutations in protease drug resistance. Journal of Biomolecular Structure and Dynamics 1–12. https://doi.org/10.1080/07391102.2017.1417160

Wong, C.-F., Yeo, J.Y., Gan, S.K.-E., 2016. APD Colony Counter App: Using Watershed Algorithm for improved colony counting. Nat Methods Application Notes.