emap Labs: Basque retrieval benchmark
emap Labs is an open benchmark for geographic retrieval in Basque and Spanish, with DOI 10.5281/zenodo.21282784. If the retriever invents an answer, it fails.
The data and AI layer behind emap, published with a DOI. A golden corpus of 139 hyperlocal searches across 22 layers, a strict held-out split and abstention cases: if the retriever invents an answer, it fails.
The finding I did not expect: the Basque gap was not about the language, it was about the model. With MiniLM the semantic stage barely contributed in Basque; switching to multilingual-e5-large raised held-out Spanish from 58% to 73% and Basque from 63% to 71%. The Basque in the corpus is written, then checked against Itzuli, the Basque Government's translator — not unreviewed machine translation.
- Citable DOI: 10.5281/zenodo.21282784
- 22 layers and 27,515 points of interest, versioned
- First hyperlocal mobility MCP server, with 5 tools
- Evals in CI with regression gates on every push