Modelling virtual worlds for scientific research

Com­put­er graph­ics can be used to rep­re­sent ani­mat­ed vir­tu­al spaces in three dimen­sions. Marie-Paule Cani car­ries out her research in the Mod­el­ing Sim­u­la­tion and Learn­ing divi­sion of École poly­tech­nique’s Com­put­er Sci­ence Lab­o­ra­to­ry (LIX). A renowned researcher who has won awards for her work, she explains: “Orig­i­nal­ly, LIX was main­ly a fun­da­men­tal com­put­ing lab­o­ra­to­ry. In oth­er words, the researchers worked main­ly on the math­e­mat­i­cal and algo­rith­mic foun­da­tions of our dis­ci­pline. Then the lab­o­ra­to­ry diver­si­fied and now cov­ers a wide range of themes, from bioin­for­mat­ics to arti­fi­cial intel­li­gence. Since 2017, sev­er­al com­put­er graph­ics teams have devel­oped, includ­ing the VISTA team, of which I am a mem­ber. On the one hand, we are work­ing on new meth­ods for aid­ing cre­ation, based on knowl­edge and/or machine learn­ing, as well as on meth­ods for ani­mat­ing move­ments and defor­ma­tions in the 3D vir­tu­al worlds thus created.”

The appli­ca­tions of com­put­er graph­ics are many and var­ied. It helps to pro­to­type and vir­tu­al­ly test objects that are to be man­u­fac­tured. In the spe­cial effects sec­tor, it can be used to cre­ate spec­tac­u­lar scenes for cin­e­ma and ani­mat­ed films. Video games, anoth­er key area, use these tech­nolo­gies to plunge play­ers into immer­sive worlds that are visu­al­ly as close to real­i­ty as pos­si­ble. Final­ly, 3D vir­tu­al worlds, which can be explored immer­sive­ly, are essen­tial for train­ing sim­u­la­tors in high-risk sit­u­a­tions, what­ev­er the field (trans­port, med­ical, ener­gy, mil­i­tary, etc.).

“For some years now, my main research project has been to explore the use of com­put­er graph­ics as a medi­um for visu­al rep­re­sen­ta­tion and exper­i­men­ta­tion for sci­en­tists in oth­er dis­ci­plines.” When we think, we have visions, ani­mat­ed rep­re­sen­ta­tions of what we imag­ine. The researcher illus­trates her point with the exam­ple of a cell biol­o­gy researcher. “Using their knowl­edge, they men­tal­ly visu­alise the process of cell divi­sion, for exam­ple dur­ing the growth of a tumour. How­ev­er, the schemat­ic rep­re­sen­ta­tions that they can cre­ate with paper and pen­cil will only show this process at a fixed moment in time.” If they want to gen­er­ate 3D ani­ma­tions, they will have to explain their vision to an artist who has mas­tered mod­el­ling soft­ware. “After a lot of to-ing and fro-ing, they will prob­a­bly end up with an illus­tra­tion that comes close to their vision. How­ev­er, they will not be able to change the para­me­ters or inter­act with the mod­el represented.”

“Our aim is to pro­vide sci­en­tists with the means to cre­ate 3D and ani­mat­ed illus­tra­tions them­selves, based on an “expres­sive” mod­el­ling method­ol­o­gy, the foun­da­tions of which I helped to lay. It is inspired by the way we humans under­stand the world around us and cre­ate objects in it.” The result­ing inter­ac­tive vir­tu­al worlds are very rich, and can serve as a sup­port for sci­en­tif­ic thought for researchers from many disciplines.

Nature, with all its rich detail, is dif­fi­cult to mod­el. A mov­ing head of hair, a water­fall or a wind-blown for­est are all com­plex phe­nom­e­na that are dif­fi­cult to ani­mate in real time. To meet the rep­re­sen­ta­tion needs of sci­en­tists, the VISTA team has devel­oped a three-stage methodology:

First, it uses mul­ti-lay­er mod­els, i.e. it breaks down the prob­lem into dif­fer­ent sub-mod­els. “The sky is par­tic­u­lar­ly dif­fi­cult to ani­mate,” explains the researcher. “So, to rep­re­sent clouds quick­ly, we com­bined sub-mod­els. First, we stacked 2D lay­ers rep­re­sent­ing dif­fer­ent cloud lay­ers. Then we had to inte­grate the air flows. To do this, we com­bined the mod­els with atmos­pher­ic dynam­ics – i.e. trans­fers between lay­ers – mod­elled using flu­id mechan­ics. Final­ly, to get as close to real­i­ty as pos­si­ble, we clas­si­fied the dif­fer­ent types of clouds (stra­tus, sir­rus, cumu­lus, etc.) and added pro­ce­dur­al details to enhance visu­al real­ism. The result is an ani­mat­ed com­put­er-gen­er­at­ed image that rep­re­sents the move­ment of clouds accord­ing to their nature and tem­per­a­ture. Final­ly, even very com­plex objects can be ani­mat­ed approx­i­mate­ly in real time.”

The sec­ond ingre­di­ent in cre­at­ing visu­al rep­re­sen­ta­tions is expres­sive mod­el­ling. 3D mod­el­ling soft­ware is often dif­fi­cult to learn. Marie-Paule Cani and her team want to make it pos­si­ble to cre­ate direct­ly through ges­ture. “The idea is to use these expres­sive ges­tures, which resem­ble draw­ing and sculpt­ing ges­tures, to enable the sci­en­tist to cre­ate an ani­mat­ed 3D envi­ron­ment that cor­re­sponds to what they imagine.”

Final­ly, the method­ol­o­gy makes use of learn­ing from exam­ples, be they ad hoc process­es, deep learn­ing or rein­force­ment learn­ing. These process­es make it pos­si­ble to cre­ate con­tent (shapes or move­ments) that is indis­tin­guish­able from the results of sim­u­la­tions or exam­ples pro­vid­ed by the user. When­ev­er pos­si­ble, ad hoc process­es are used, as they are less cost­ly in terms of time and ener­gy than deep learn­ing – where the machine must ingest a large amount of data. “To cre­ate a land­scape,” illus­trates Marie-Paule Cani, “the user will place cer­tain ele­ments (peb­bles, trees, tufts of grass, etc.) in a small area by hand, to illus­trate the spa­tial rela­tion­ships they want to see between these ele­ments and with oth­er fac­tors, such as the slope of the land. Then the com­put­er will sta­tis­ti­cal­ly learn the dis­tri­b­u­tion, i.e. the cor­re­la­tions between types of ele­ments”. Just as when paint­ing a pic­ture, the user then has brush­es with which to “sta­tis­ti­cal­ly paint” the vir­tu­al world in real time.

It is by com­bin­ing these three para­me­ters (expres­sive mod­el­ling, mul­ti-lay­er­ing and learn­ing from exam­ples) that the team can cre­ate and ani­mate com­plex and var­ied vir­tu­al envi­ron­ments in 3D. In col­lab­o­ra­tion with researchers from oth­er dis­ci­plines, they have been able to rep­re­sent the for­ma­tion of land by glacial ero­sion¹, a Mediter­ranean or Alpine ecosys­tem² or the for­ma­tion of moun­tain ranges³.

In order to cre­ate new forms, we need to be able to inter­act with the mod­els, so that they respond in real time to the user’s ges­tures. In addi­tion to the mod­els and algo­rithms, Marie-Paule Cani and her team some­times have to build spe­cif­ic tools to cap­ture move­ments, such as an extend­ed mouse with force sen­sors⁴. “We devel­oped a hand nav­i­ga­tor, a tool that con­nects a mouse with six degrees of free­dom to small sen­sors acti­vat­ed by the user’s fin­gers. This tool was used to ani­mate a vir­tu­al hand, capa­ble of sculpt­ing 3D mod­el­ling clay.” The user can mod­el live on the com­put­er, but in some cas­es, tac­tile inter­ac­tions and vir­tu­al real­i­ty devices enhance intu­itive inter­ac­tion with mod­els, facil­i­tat­ing cre­ation and allow­ing users to manip­u­late com­plex envi­ron­ments in a nat­ur­al way. She con­tin­ues, “invent­ing these vir­tu­al real­i­ty tools has some­times been nec­es­sary to fill a gap and advance research. Unlike gen­er­a­tive AI (often devel­oped to cre­ate for us), the intel­li­gent sys­tems I am inter­est­ed in allow ges­tur­al inter­ac­tion, and are intend­ed to make us humans more creative.”

The VISTA team reg­u­lar­ly works with sci­en­tists from oth­er dis­ci­plines. In col­lab­o­ra­tion with the lat­ter, the researchers devel­op cre­ation tools based on their mod­els, which enable them to cre­ate exam­ples direct­ly. “From 2017 to 2021, for exam­ple, we worked with palaeon­tol­o­gists who want to “see” their palaeo­cli­mat­ic mod­el and the result­ing dis­tri­b­u­tion of fau­na and flo­ra in the Tau­tavel val­ley.” Locat­ed in the Pyrénées-Ori­en­tales region, the Tau­tavel val­ley is a major pre­his­toric site. Archae­o­log­i­cal digs there have unearthed numer­ous bones, tools and traces dat­ing back more than 300,000 years. “The aim was to repro­duce the ecosys­tem based on the data and hypothe­ses of the researchers. We rep­re­sent­ed the palaeo­cli­mate, and gen­er­at­ed the flo­ra and fau­na that inhab­it­ed the val­ley at that time accord­ing to the para­me­ters (tem­per­a­ture, hydra­tion and sun­shine) as well as the prob­a­ble list of species present giv­en by the palaeon­tol­o­gists.” Thanks to the ani­mat­ed 3D rep­re­sen­ta­tion con­struct­ed in this way, researchers can see the flaws in their mod­els and adjust their hypotheses.

Could com­put­er graph­ics be the first step towards time trav­el? With­out going that far, there is no doubt that this dis­ci­pline trans­ports us. Imag­ine tak­ing a vir­tu­al walk through a 3D recon­struc­tion of a val­ley in the Palae­olith­ic peri­od – it’s a form of time trav­el. What is cer­tain is that com­put­er graph­ics can help sci­en­tists to improve their men­tal vision and their mod­els, by com­bin­ing knowl­edge and dig­i­tal learning.

The con­stant evo­lu­tion of com­put­er graph­ics offers fas­ci­nat­ing prospects for the future

But it also works the oth­er way round. To test their mod­el, Marie-Paule Cani’s team is repro­duc­ing actu­al land­scapes. Com­par­ing their results with the exist­ing land­scape pro­vid­ed by satel­lite images enables them to adjust their method­ol­o­gy if nec­es­sary to obtain the most accu­rate rep­re­sen­ta­tion of real­i­ty pos­si­ble, and then to val­i­date their model.

The con­stant evo­lu­tion of com­put­er graph­ics offers fas­ci­nat­ing prospects for the future. By enabling experts to visu­alise their men­tal rep­re­sen­ta­tions and test their sci­en­tif­ic hypothe­ses more quick­ly, it opens up new avenues for research and education.

One future project involves using expres­sive mod­el­ling to raise pub­lic aware­ness of cli­mate change. By allow­ing users to manip­u­late hypothe­ses and visu­alise in 3D the effects of their choic­es on our plan­et, this approach would help to get the gen­er­al pub­lic more involved in cur­rent envi­ron­men­tal issues. In this way, com­put­er graph­ics not only mod­els real­i­ty, it also becomes a cru­cial tool for sup­port­ing sci­en­tif­ic research and enabling every­one, stu­dents and the gen­er­al pub­lic alike, to gain a bet­ter under­stand­ing of our world.

Loraine Odot