Extracting Published Discussions of 13,616 Galaxies with Language Models

summary

The gist

Extracting Published Discussions of 13,616 Galaxies with Language Models addresses the difficulty astronomers face in determining whether an object has been genuinely discussed in literature rather…

In short

Language models were used to analyze arXiv papers to find discussions about 13,616 individual galaxies mentioned in literature rather than just catalog entries. This created GalaxyWiki, a resource allowing users to find and read these specific scientific discussions. The method demonstrates how AI can make complex literature analysis feasible for astronomy research.

Key concepts

Language Model Agents
These are AI programs (like GPT-5.54) harnessed to read papers and perform specific tasks, such as identifying mentioned galaxies and extracting relevant quotes. They work by being given instructions to search for specific information within large amounts of text, making the extraction process automated.
GalaxyWiki
This is a new database created from the research. It links galaxies to the literature that discusses them, allowing users to easily find what has been said about a galaxy and follow links back to the original source papers. It serves as a tool for tracking scientific discussions.
Coordinate Resolution and Grouping
This process involved matching galaxy identifiers (like NGC numbers) against external databases (NED) to get precise coordinates. Mentions without direct matches were then resolved by having AI search the paper for compatible identifiers, ultimately grouping nearby positions to define individual galaxies.

Terminology used across episodes

This episode discusses

The paper

Extracting Published Discussions of 13,616 Galaxies with Language Models · Read on arXiv

Nolan Koblischke, Jo Bovy

Department of Astronomy and Astrophysics · Dunlap Institute for Astronomy & Astrophysics

Astronomers often have difficulty determining whether an object, such as a star or galaxy of interest, has been discussed in the literature. Existing tools enable object identification and linking in papers but do not record whether an object was genuinely discussed in a paper or just appears in a table or catalog. We address this by using language model agents to identify galaxies discussed in more than two sentences across a corpus of 156,006 arXiv astro-ph papers and extract those discussions. We extract 44,332 discussions of individual galaxies containing 85,314 quotes from 18,641 papers. With sky coordinates for 98.2% of discussions, we group them into 13,616 galaxies. We bring these results together into GalaxyWiki, where users can find literature discussing galaxies, read what was said about them, and follow links to the source papers. The code, dataset, and webpage are available at https://github.com/NolanKoblischke/galaxywiki-paper

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: I'm Vera.

Jocelyn: And I'm Jocelyn, here with Subrahmanyan.

Vera: We're looking at the paper: "Extracting Published Discussions of 13,616 Galaxies with Language Models".

Jocelyn: Here's the gist of the paper. Extracting Published Discussions of 13,616 Galaxies with Language Models addresses the difficulty astronomers face in determining whether an object has been genuinely discussed in literature rather than just appearing in a catalog by using language model agents to identify galaxies discussed in more than two sentences across a large corpus of arXiv papers and extract those discussions.

Vera: This method results in the creation of GalaxyWiki, a resource that allows users to find literature discussing galaxies, read what was said about them, and follow links to the source papers.

Jocelyn: Data Acquisition and Extraction Process The source corpus utilized is preprocessed arXiv paper text from the MINT-1T dataset, which contains 156,006 astro-ph papers from June 1993 to May 2024.

Vera: A filter was applied initially to retain only papers containing the standalone word “galaxy” or “galaxies,” leaving 87,899 papers.

Title and authors: Vera: So this paper, "Extracting Published Discussions of thirteen thousand six hundred sixteen Galaxies with Language Models," it tackles that real problem astronomers have—figuring out if an object has actually been talked about in the literature or just listed in a catalog <ref:2611.13118#pg1,Extracting Published Discussions of 13,616 Galaxies with Language Models>.

Jocelyn: Right, it's not just finding a galaxy name; it's digging into the actual sentences where people discuss that galaxy. The authors use language model agents to find those discussions across a huge collection of fifteen thousand six hundred six astro-ph papers and pull out quotes that are more than two sentences long.

Subrahmanyan: So, the goal is to move beyond just knowing a galaxy exists in a table and understanding what the actual scientific community has said about it. It sets up this new resource called GalaxyWiki where you can find literature discussing galaxies, read what was said about them, and even follow links back to the source papers.

Vera: That sounds incredibly useful for anyone trying to see how different pieces of literature connect on a specific object like a galaxy. The process they used involves having these agents look for one to three quotes, totaling three to eight sentences, along with a summary supported by those quotes.

Jocelyn: And the initial extraction step yielded forty-four thousand three hundred thirty-two discussions of individual galaxies, which came from eighty-five thousand three hundred fourteen quotes across eighteen thousand six hundred forty-one unique papers. That's a lot of data to process.

Subrahmanyan: The scale is impressive, but the next challenge they had was getting accurate sky coordinates for all those galaxies so they could group them properly. They started by using identifiers that are standard, like NGC or SDSS or two mass identifiers to look up positions on the NED database.

Vera: And that programmatic query against the NED API worked well for twenty-three thousand seven hundred seventy of the twenty-five thousand two hundred six attempted identifier queries, which got coordinates for about fifty three point six percent of all mentions.

Title and authors: Jocelyn: For what wasn't found in those standard formats, they had the language model agents search the original paper and then query the NED API using any compatible galaxy identifier they could find. That approach ended up giving them coordinates for six thousand seven hundred one records from source papers, eleven thousand five hundred three through NED, and one thousand six hundred eighty four from external internet sources.

Subrahmanyan: Those coordinate results gave them information for forty-three thousand five hundred forty of all the mentions they were tracking, which they then used to group positions that are less than one arcsecond apart and included neighbors of neighbors to get down to thirteen thousand six hundred sixteen entries interpreted as individual galaxies discussed in the papers.

Vera: The accuracy check on those extracted quotations was pretty rigorous; they compared each quote against the source paper text and found character-for-character matches for sixty-six thousand three hundred sixty-two out of eighty-five thousand three hundred fourteen quotes, which is seventy seven point eight percent.

Jocelyn: They also normalized the Unicode and spacing, which brought that number up to one hundred fifty-nine thousand seven hundred seventeen matches, making the total match count eighty-two thousand three hundred thirty-three at ninety six point five percent.

Subrahmanyan: Even after ignoring case and removing all characters except letters and numbers, they found another thirty four matches, bringing the final total of verified quotes to eighty-two thousand three hundred thirty-three at ninety six point five percent.

Vera: They also did a manual review on one hundred twenty distinct galaxy mention records, and they found support for the intended galaxy identification in fifty eight cases. It’s good to see that human eyes can still help catch some of the tricky stuff.

Jocelyn: Despite some issues like truncated identifiers or rounded coordinates reducing precision in three otherwise correct records, their review supported the conclusion that the pipeline correctly identified an individual galaxy and extracted its substantive discussion in one hundred eighteen out of one hundred twenty cases.

Subrahmanyan: This GalaxyWiki dataset is presented as a unique resource for figuring out if galaxies discovered by anomaly detection methods have been previously discussed in literature, and it's also meant to help enhance descriptions for training an astronomy specific vision language model. That kind of database could guide similarity and semantic searches for galaxies with related features or support literature surveys in astronomy.

Title and authors: Vera: It really brings together the scientific interpretations of galaxies and what makes them noteworthy, allowing astronomers and the general public to check if a galaxy has been discussed in the extracted literature, read those discussions, and continue to the relevant papers.

Jocelyn: The methodology itself is interesting because it shows how language models can make previously impractical scientific projects feasible by performing tasks that can be done in parallel and verified. It’s a nice demonstration of agentic workflows on this scale.

Subrahmanyan: I think the potential here extends beyond just galaxies, because this kind of approach could potentially be extended to stars, supernovae, exoplanets, and even theories of dark matter. Future work will focus on expanding that corpus and evaluating the mention identification and coordinate resolution to build a more complete collection.

Vera: So, in closing with "Extracting Published Discussions of thirteen thousand six hundred sixteen Galaxies with Language Models," GalaxyWiki is this database that brings together scientific interpretations of galaxies and discussions of what makes them noteworthy <ref:2611.13118#pg1,Extracting Published Discussions of 13,616 Galaxies with Language Models>. It lets astronomers and the general public check whether a galaxy has been discussed in the extracted literature, read those discussions, and continue to the relevant papers.

Jocelyn: This work illustrates how language models can make previously impractical scientific projects feasible by performing parallelizable and verifiable tasks. The limitations they mentioned include potential misses during extraction because of limits on the number of galaxies per paper, and reliance on the astro-ph subset of MINT-1T <ref:2611.13118#pg2>.

Subrahmanyan: Also, they noted that programmatic NED resolution produced false positives with Abell designations matching local identifiers, which required them to invalidate one hundred twelve rows. The work was enabled by credits from OpenAI’s ChatGPT Futures award and Google’s Gemini Academic Program. This paper also used the NASA/IPAC Extragalactic Database and the SIMBAD database.

Vera: It's a really solid piece of work showing how you can use these models to do things that were previously impractical, like systematically analyzing massive literature on objects in space. We'll be back after a short break.

The paper's summary: Vera: So, this paper is about taking those huge amounts of literature—think fifteen thousand six hundred six papers from arXiv—and using language models to figure out which specific galaxies people actually talked about in those papers.

Jocelyn: Right, it’s not just finding a catalog entry; it’s digging into the actual text, looking for discussions that are longer than just a few sentences.

Subrahmanyan: The authors built this GalaxyWiki resource so people can check if a galaxy has been discussed, read what was said about it, and find the original papers. It connects the dots between discovery and scientific talk.

Vera: They did this by having AI agents look for quotes of up to three sentences long, plus a summary of those quotes. The result is that they got forty thousand four hundred thirty-two individual discussions from eighty-five thousand quotes across eighteen thousand papers.

Jocelyn: That’s a lot of material, but the real trick they used was matching those galaxies to sky coordinates using standard identifiers like NGC or SDSS.

Subrahmanyan: They had programmatic queries against the NED database, and that worked for about fifty three point six percent of all their potential galaxy mentions.

Vera: For the ones that didn't match standard names, the language models searched the original paper and tried to find coordinates using any compatible identifier they could spot. This gave them coordinates for forty-three thousand five hundred forty of all those mentions.

Jocelyn: Then they grouped everything by linking positions that are less than one arcsecond apart, which ended up giving them thirteen thousand six hundred sixteen entries as individual galaxies discussed in the papers.

Subrahmanyan: The verification step was pretty detailed; they compared each quote against the source text and found matches for about ninety six point five percent of all quotes after they did some cleanup on the text.

Vera: They also did a manual review on one hundred twenty records, and that confirmed their identification in fifty eight cases, even with some messy input like truncated names or rounded coordinates.

Jocelyn: So what this means is that we can build a tool to see if new discoveries have already been talked about in the literature before we spend years studying them.

Subrahmanyan: It’s a way to enhance descriptions for training vision language models because you get the actual scientific context attached to each object.

Vera: The implication is that this resource can guide similarity searches for galaxies with similar features or help structure literature surveys in astronomy.

Jocelyn: They also mentioned a limitation—that the method might miss some discussions if a paper only talks about too few galaxies, and they had to toss out some entries because the coordinate resolution gave them false positives matching other things.

Subrahmanyan: It shows how language models can handle these parallelizable tasks that were previously too hard for researchers to do manually at this scale. We could potentially apply this same approach to stars or even dark matter theories next.

The paper's improvements: Tom: So, we just talked about how they used language models to find discussions in literature about galaxies and got thirteen thousand six hundred sixteen entries for GalaxyWiki.

Vera: Right, that was the initial setup using agents to pull out those long quotes and then using standard identifiers like NGC or SDSS to get coordinates.

Jocelyn: And we covered how they verified those results by checking character-for-character matches on the extracted quotes, getting ninety six point five percent accuracy.

Subrahmanyan: The paper is suggesting ways to make this system better, mainly focusing on improving the identification of those galaxies and cleaning up the coordinate matching.

Vera: They are proposing an improved literature tracking system where you can immediately check if a galaxy has been discussed, read what was said about it, and see exactly which papers contain that discussion.

Jocelyn: That means instead of just getting a list of quotes, you get direct links to the source papers right from the search results.

Subrahmanyan: That would be useful for tracking an object’s scientific history in real-time, which is vital for structure formation studies where context matters a lot.

Vera: They also suggest improving how they handle coordinate resolution by flagging cases where identifiers are truncated or where the positional coordinates were rounded, like losing up to twelve arcseconds of precision.

Jocelyn: That means the AI agents need to be smarter about spotting those messy inputs and not just blindly trusting a rounded number from a paper.

Subrahmanyan: From a theoretical standpoint, if we can clean up these mentions, it helps us build better knowledge graphs connecting objects across different papers.

Vera: It also implies that the next step for this GalaxyWiki is to expand the corpus significantly and evaluate how well the mention identification works on a much larger scale.

Jocelyn: They are pointing toward using this resource to train astronomy specific vision language models, which is a big deal for interpreting images of galaxies.

Subrahmanyan: Ultimately, refining these methods means we get a more reliable map of how scientific ideas evolve around specific cosmic structures.

Conclusion: Vera: So, we’re wrapping up this look at "Extracting Published Discussions of thirteen thousand six hundred sixteen Galaxies with Language Models." We saw how language models can systematically dig through massive paper collections to find specific discussions about galaxies.

Jocelyn: It really shows that these tools can act as a massive literature search engine tailored for astronomy, which is something we’ve been hoping for in terms of connecting observational results to the broader scientific conversation.

Subrahmanyan: I think this process of linking discoveries to literature is fundamental because it builds the context needed to understand how structure forms over cosmic time.

Vera: And while they got a solid dataset, they also highlighted that the next big step is improving how they handle those messy inputs, like truncated identifiers or rounded coordinates.

Jocelyn: They’re focusing on making the AI agents better at cleaning up those errors so the results are more reliable for actual scientific use.

Subrahmanyan: I think refining that coordinate resolution is important because we need precise locations to connect these textual discussions to the actual sky positions we measure.

Vera: Exactly, and they mentioned that this GalaxyWiki resource could be used to train vision language models for astronomy, which is a huge potential application.

Jocelyn: It means researchers can search for galaxies based on their features and immediately see what the literature says about them without having to manually sift through thousands of papers.

Subrahmanyan: That capability would definitely help in mapping out how different theoretical predictions are supported or contradicted by existing observational data.

Vera: So, this paper demonstrates that language models can handle these complex, multi-step tasks—extraction and resolution—in a way that was previously impractical for researchers to do manually.

Jocelyn: It’s a good demonstration of how parallel processing can make scientific literature analysis feasible on such a massive scale.

Subrahmanyan: We need to keep pushing this idea toward things like exoplanets or dark matter theories, because the methodology itself is quite versatile.

More episodes

← Home