Lithuanian researcher created AI-based solution for measuring the emotional climate of New York

A young researcher from Kaunas University of Technology (KTU), Lithuania – Domantas Didziapetris – created an artificial intelligence based solution for measuring the emotional climate in Manhattan Island, New York. After carrying out a sentiment analysis of over 36 thousand tweets, he created a scale which indicates how the residents and visitors rate emotional climate of the different Manhattan neighbourhoods.

New York is one of the world’s most-visited cities. According to the official data, in 2018, New York City welcomed a record 65.2 million visitors, and the numbers are constantly growing. Manhattan, the most densely populated of the five boroughs of New York City, has a population of 2.7 million and attracts almost 15 million tourists annually.

“Manhattan is the heart of New York City, bursting with life. And what could be a better place to test a new methodology if not the city which literally never sleeps?”, says Domantas Didziapetris from KTU Faculty of Architecture and Civil Engineering.

According to the young researcher, urban analysis methods often still use paper-based or online surveys. Yet they are not always reliable, and the results may not represent the actual situation.

“One glance at a social media platform may provide more insight into the actual situation than an in-depth questionnaire. Moreover, the opinions expressed here are far more expressive and impulsive, as people tend to express joy or disappointment online the very moment they feel it. One of the best platforms to gather such data is Twitter as it limits the number of characters in a message, it is easy to depersonalize the data and the agreement to share the data is already granted once a user signs up to Twitter”, explains Didziapetris.

A couple of IT programs were written in order to process the data. Firstly, Python was used to compile software that would gather the tweets in real time. In the period of around 4 months, more than 1 million tweets were collected, 65.447 of them had geographical coordinates. After deleting all the entries, which were from outside of Manhattan, 36.543 tweets were analysed further, classified by the community district boundaries as identified in the New York City Open Data portal.

Secondly, the researcher created a program for sentiment analysis of the tweets. Two criteria were considered – subjectivity and polarity. The subjectivity indicates the factual content of the tweet and the polarity – its emotional tone. Both of the criteria can be rated from -1 to +1.

“The higher the subjectivity criteria, the less reliable is the entry. The closer the polarity is to -1, the less positive is the emotional tone of the tweet”, explains Didziapetris.

The sentiment analysis results were projected on the map, and, according to the young researcher, this clearly revealed which neighbourhoods are, so to speak, happy, and which ones – not so much. Upper West Side had the least favourable emotional climate in Manhattan.

Upper West Side is still too big of an area to draw conclusions; therefore, it was necessary to pinpoint the actual location which would need reviewing and urban development.

To complete this task, the “Urban Network Analysis Toolbox for ArcGIS”, a toolkit created by MIT in 2011, for urban network analysis was used. One of its methods – betweenness – is typically used to calculate and to estimate the potential of passer-by’s in the network. The more reachable the area is, the warmer (red) is its colour on the map.

“The less reachable the area, the more abandoned it is, and the abandoned areas are usually more prone to criminal activity. After urban network analysis, one of the areas in the neighbourhood looked especially suspicious – after the visualization, it turned blue and green. My hypothesis was proven right after I received the photos from the area. Simply put, it lacks safety. My supervisor, who took the photos, described the common feeling in the area as ‘unsafe’, ‘not cosy’, ‘encouraging to leave as soon as possible’, ‘old’. The really interesting thing is that this territory is nearby active and reachable buildings in the East, such as Lincoln Center, The Juilliard School”, says Didziapetris.

The research, which was presented in Domantas Didziapetris Master’s Project, was awarded by Hnit-Baltic company, Esri Inc. representative in the Baltic States. Domantas Didziapetris won an opportunity to present his research to the international 18 thousand people audience of Esri User Conference 2019, which will take place in San Diego, California in July.