{"id":174,"slug":"ntu-nlp-sg--xcodeeval","name":"xCodeEval","author":"NTU-NLP-sg","description":"The ability to solve problems is a hallmark of intelligence and has been an enduring goal in AI. AI systems that can create programs as solutions to problems or assist developers in writing programs can increase productivity and make programming more accessible. Recently, pre-trained large language models have shown impressive abilities in generating new codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. However, the evaluation of these models has often been performed in a scattered way on only one or two specific tasks, in a few languages, at a partial granularity (e.g., function) level and in many cases without proper training data. Even more concerning is that in most cases the evaluation of generated codes has been done in terms of mere lexical overlap rather than actual execution whereas semantic similarity (or equivalence) of two code segments depends only on their ``execution similarity'', i.e., being able to get the same output for a given input.","tags":"[\"Task_categories:translation\",\"Task_categories:token-Classification\",\"Task_categories:text-Retrieval\",\"Task_categories:text-Generation\",\"Task_categories:text-Classification\",\"Task_categories:feature-Extraction\"]","license":null,"framework":null,"parameters":null,"downloads":653983,"likes":77,"verified":1,"created_at":"2026-04-20 14:59:20","updated_at":"2026-05-08 16:45:14","source_url":"https://huggingface.co/datasets/NTU-NLP-sg/xCodeEval","source_platform":"huggingface","hf_repo_id":"NTU-NLP-sg/xCodeEval","ollama_name":"","category":"dataset","latest_version":"v1.0.0","version_count":1,"signature_count":2,"risk_level":null,"risk_score":null,"versions":[{"id":173,"model_id":174,"version":"v1.0.0","manifest_hash":"0fd7288672dae3d53d19b7c4737d92e9611397b76d4fd969392aaaa62fb7fab7","file_count":0,"total_size":0,"r2_manifest_key":"manifests/datasets/ntu-nlp-sg--xcodeeval/v1.0.0.json","created_at":"2026-04-20 14:59:20"}],"files":[],"signatures":[{"id":535,"version_id":173,"signer_did":"did:quantamrkt:registry:shield-v1","algorithm":"ML-DSA-65","signature_hex":"73c34afa223f3491243860319155ee59bb5f9dd58d5b959e95f78ffd8bd93a0d","attestation_type":"registry","signed_at":"2026-04-20 14:59:20"},{"id":697,"version_id":173,"signer_did":"did:web:quantamrkt.com:chain:authority","algorithm":"ML-DSA-87","signature_hex":"d645f38c9fdabc3611edd8fda863aa3d64dfb672437d29efb5cbe07c6accb6d2f135562ece2253f6bc6e432759da239fd9fb7e383e1708049d633cfe84a2e90a9044ee3de01edd09ee1bcd3809786a31513495ba38b40d6772aa48bfe86eb0446b3f522717a3ef97c1943ff404fb2ea47ac4649bd71a3854fbe575fb0b21b45c8487ee0061b27fd1074aef039ab142b56f6fbfd928dc69469d48dc3c627746fa5772bf82d65cc5105f313c0a9427b2f2c4ad51383b1836cd57ec650d026a3a822db5d382bfb6578be2f3e532b2ee73c0f61f6e5eb3ea8a6912032bf944219276eaaa30530b64de6cc242ab5eae4a23c2193e812c0697127df31ae9281ce94ed1c8a3df6c696a86a9fe5205bfd0edef3029ade172611b4206ab619a5f3a06e2d97b30688c1f8dc55239e1823e47e9a292f305df444dc5be059f3d43c0a06f39fcb153293bb265fbb1f677fa65b4f8aea438606ecca75f2cdfdce5cc15d9709f824ecf35b2ca4294616c6ee4b50f643d0b3fcdf125b396af222f58c0548c79036c9e35db44cb83f0dcaf6eb6be3131d72581670037ccdfaa3fbce38ed5a5e2eafadd9f09ddf2db7c05e97b41495382f604b0dd6f8c3c2aa741d802b404350df73b2941a6050d3749550ef26aa1011e0295a8c43f171b25875f216e28524272e2ae94ffd6eb3c4c1cdd1ec123f54b3936c1edf3385bf6eaac6e64af4c382633ae5b0c4f6a10e2c408cacc17862f380100c8bf34ca03768c2516b268df01eba3ce6fa9d49c3162bfef9ba54f5b7d2bd81da04850fa61ea660a84c4255bfd16e95a4455e5054eeb0e18e22b38891c405ce4f0d646f47e22e49ef1db4ffed63c29653e7b13a0036a817008a2c1fb097edd3ad9bcc04f676e3fa0f8455f96a95aa15b0a4fd5c62053ef83ecdfebe35aadf04e8ddc1e6570089b11db6cf476b4d492a794b387fe7854bbc1a287687f675d5a8951285d78392188aab573fc3ebb3c2c454366ebd023fd525ce039c8f5784935f83c30ed37c5f06dbad75451bb6bb3d667a78b12a77e89a066adbd990e926f644f41cf92f74d5009a8d9f7afcae4972add0b4d3833f8020f98078bb003a23fa975a0164705f26d8d9d3ab57f541a42cec58325929f118446ede5896081a4221673a615c80213a47314acae88b48e92b4de7857f80303d13c55b9a4bb5911dfd16a0cc3f46ad6e163c4a7d43d4962642674abf014dd157ad3393f2cb5d7c5d93b4019267e202c8cbbd356e88cef6955d25e14e208e03c83a1d5267db0f251f9f21b09559372573d3a0101ce152aebc42b0f5e8a124425c633c55bb6b5dd2aacb5d72f0dcf55a2ceb4f5cd6c3e8fe23d3ed84d31fd0c8e8bfdd5167095742f0360dcdc669ea341755237981d2a7634d4df0639b69ca7e7f084a25b11569e5ae9d9ae6b7f9e05b709003b7e1baa08c907bb69e4783657a26618c8bc5c7d50dfba814d6f3de718bb3bccd4fb2526e4ab368562773596b2900aaa1cc0a435351fce20a148defd4841a3288f36d6221d98a9c08af315d7550f09f2aea830cbddb4ee2fe9f7ac99d91a83504db39b1766c8e39cd0bba642741c2ebf280ad3e186f37f950b7b804df0a5eb501305ebd0a72c6fda588c85c8a85660e05107e4dacf4b80816089bb3e4a50f3078114c75dfc316849164c4629407317f31fe50ad495605e7568c8073cf7bef1aa50df82da81010954f045ae9371aa795ef51e0af5f2212d37b45810469553f7631c833a36ae0f479f67789c8a1f9598c8961ae480b2743c54de1dc579c1ac5340e3573a62bc8cffc857d8843e1d8c6853dd9c1ca7f06720627cbf8be648d197ccfc3aadf742063afac13143803ee5ffd170cf42bd136fd6b531f3a195357e7a1b714569000a7f5c8709570e1e567eb1275efce6163c6a25ecdc9f81d79fe80c1e4a5c1e24aeb5329df6da9cf593d8f1ad05adae44f5b76621974719c70911a8d78adbcd1be19327027bbad9e3e57c09e961629631248b04788e2f73d6ace1b7803011c9ef9527ff9b9ceabf10dd2f8af0ab1cf97ab1221778663e399bf4e26d81dd97a5f6519c86a87c166a181b65695275f329ce9ddb08a43e8b4dcd5c17142800186c153a86bbe848b8985ad4f995d754ba0165f1afe58656b79b36bdde25118103cb783a405af4a16d34c580e4c71be8618c80d6ab9d3f8c7bb017c7bf0a104acc66bd0bd2290f50d0b75cdfec3db99ee307e38c0027df10fa6044401e315646cf56c7a57d94fcf4f9c74c43d0d463a9c40466a517f61c56050ef6822e78d4b25893653196a37b42dd2e9efd9210ac4630dddba329335a72a7b97f9df8b22cc976d8a7f48a51436215adea4fe905737810eb84be3c6bb3a1f160b27d517b9aaa62315c0571213d67a5bb2f4727bf1ae323ad4793568387b4a045673ee861758e8619cb4cb7ca7f9465255d4afc98dc8ec7f28aaa06a91856d126d3c642b576745749e6a510677504c1289209ddf4097234cc0c8227bf071262010660fadb05183d57beaf3584569d3aef76cd8e4604db05b1ca0452677673ba633c24fdbb21bf5d7b1283faf501e9b97910eeef4a4c5358cccc674e8de11f436fd26b6eeb79445b78866ce464e3b4bdc1b15479d47053dd8b7639bfd8b2e7a2515a3fa00b08b348d5a1175e1d5a041ef0335161d688b729c92adc0f87a9deb983a9532960d007c10d500b7ea5dad91cb7ddbd0222af67127a39422f116b75646ea93415a9144ac409904230b15904de2aec90b945c55b20d59710c59156f1e35ab8de91168176d9c17afb8f7b707db411eedce80e8e3a920139b4ae1a276361249b1c08e8ab9040b6de50e26d139c15999ca09a7cec4794a16b7151177e4fdefc8610544828ce036d2c56d31f25ebce8e5ff2b668caa269f2d803b929dbf62eeefb403ee36e0a23f046162614eb18043b02aa722f3044524a64fbc3a16c0f4e8c747e4afcac58e0af35dd8eb35df84fc18b23d67bdc6e593add844dc0263c1b1c40140053ea62a67b8cbab90cca40163bc7bc93ad5df062636a19833ae2ebbeda87e749b8b77ee99d7c6e891cccb16eeb0221062d5d64d397c03296c5680d751027faf726c51e8cfe0618e6ad04b2caadd409d04d7df84c5bea6f98e49f6934548c0c06c6360fb23b762f1826cc166635a1b70e8dac04d0ab7ceb28116ded68aecc40d3cc730424749e54ab903dc539f7516c06bb6941d17885cec7038ded21d683f32a52d4f474114aaed4fbd377546f756e059444d38f0964ce6c697eb72e3ee2cab9e24bcbcc566c9c416197b7465307cd67c9abc8b43d43bfee8e7a568afb46e8776cadadc8a2aa8318d62c8c1fd529dda2e74c6ece7eac26b2134b2fe7c53668fd4f8e770a0bda090387a428488ad914295c3758113a15bb553c2598ffa9d20ba0346840542733337c3b4a9e55737811360096810cab392441b4a5f14dddf786e2c67469709e9c15f783f5e3c5480a64c624862c2532496331b1a9c86618061b53203e6a5a71193d4d9872e95b027b55ee439c4496b2493b4fb8ab61abee5c62b74ed911e73a5ed27d6269769ac3fa9799802b3cb8ac62457872fd9c0bfbde34d8cee914d2842c3806f8cad925dc609afdad1e2a04d55add35424eb9ebe933d1119a4dfe201d03aa8805c0cfc1379af47ed8cf93d7c2ad8838dfe5e3a2ce17667539723b571daf8ea984474675b601674a68dc93d720d8de6b0470df983096c8435d06892b260b7dd7b825606b75cf45f0b7867275ea26bc01d5eac9418f96e8a8c6a6f17b8e5b9291457ca66a9858f3b832a906b91d10680aa0d7bba9d2915d5834c34e8b03e4c2043764e5de3fa90e757b0df2cece6aec5bd85e6320c772c4233e7dcc7f5ac5c0804659ce4ce1ce65cd24fd0c4299eb75479e63fc3a8cc0549adbca3a6bc273ce3f07ea69a038aec0e34fbdc3813e10dba6db76b5607521caaf1c04e9b24d0898bc7672608c7e857061f87a0005cefe6c1310ad941fa2ab688bb1cc04081dfcddc6b3c37aaec7a2e1d6460e02662357172846c4fd0d36aa457f9726f90fd26ee8dcfaa8ca50e7ece9dc791f54d7810f8c68ebf4e232ae59d6adaad60f98432fd225ade6158bb3572aaad44dd16f3672ab6ff815544f415a3cc359e0868d5404b8d2efdd7e387e44b6522acc0f3a4007257dd6fd5eae4755f987800795bcf944190df366d57935fac16b90643786d5606328f3b9291def79245f8760aa4349789a6b26028fc0bc94d95729d22c2b3f44babdda292c6a9e96ab9d257bc6768b59b81d4ba82497e2b250a131a8fa2d20eb17540b13963cd80a1c62246ecb5db486f120386a0613bc44b58558e9b8ad7810d0090bd5c6bf2190ac3f7068750e8cee2aa11821da5cb5cb742a8ae5967ae6a031617fbc1e105512e83fe31e2fdb56d02e90a260830f2417da8d6e2249b5d8e47d8995d689fd29ae426c91f91862c5100add9d03992a340c74908c028c4860363857b6d6d6602db462a7d3ceb699ad2495dfab6b411e3fcc1ce77fa3e0c29080520a5a4d4f933451b672716199e151fa844f3e4b1a0363fd48f3167524915d3082006f415b634c6664f4b46c20ae13f09d8cedeb1f59c9d3392adc8f8c05f4f66e240bef18aba1a344a54b7e74c81a6ea411780f5f6a127d7003884b39b93d4b0dd26cdd54648dce52b81c5ea87cae6dfa3fa2d9270c0284f20fdff7a51f92d46ea2b999eef05a5ca307cf77020f67e7b942dcef554ed2acee5cbd125d7bcce599dcf28c5e56366cf05f8b33f3eef64f7d9af1ca678137968bb0661d5640de0c88535d0bc599464292ab6a466daa0a1c59f1d27c9928afc8cc86990887044f82da48cfe3e6117745d5f695171d84598607ba44128ee29663b6bb0503084f9878d6d0dfd2ea2a7feabda35b8128da40f4dd432dc18224bda13acfb3712d70b5316472befebd58b273c4907f59b4255af30a07ea34b2e5e7cc68be533a3f6ea9961dbbb3ece3154f65ea48095226a208541800f4e23609128ae0c177da4924f7e0cc79a9b50cf551a12a8297228a9898c6101fc0db6901c449037f0cbd0c2435aa92f1366f77fdfe565f30faf55b534326b8ccc47e7ab0276a2c69c519fd53f2d3e663c20d66b9353d96a6ca7173b06717deec242cab98b3e0e32907aee75c397afa56cfdeb704a750d80a3ab3199709e327da35aedab8e1171b0b1b518ae68bdf900da44e92bc2393a557b1080fd29d8cef10e585299bf253b54f219fb7ee4dd04e72951bfaee1b8a967f52360e7a15a973cd312d2b561caf7901ce0dc83b5dca942fdda9d6e6e0daf4631eac675f381bdac7f0494451336d8cb969e228a7f9b5f53ede958a2445f688908df0e34f2c01b0db22f16dea7c92d6aac384c94c9f3ba64f4916b9c3519b69e43777114182eccbdea40af99eb822f74fed7f2e9eb6cdda56a88ff2e5a4fa8bf266364a9c6086c4e5c91acf9a34c75bf8fa3b199459e7debdec65473f07ff1a36b0ba9c4f2f4e9788b9a66e2b4869e3e704a81d97e2868c426f5c9f02ed1a2aa0416218d666aa8544f2ed38e0f15c149a73fe03636910857e067f0afae14f9b257552b931ddc2748740911b05c2b9bbfb955cbf1b12f4990283a478ecdc6bfadbb7a20bf4edd6f29b6021c71801195d5330cf1b0d861e6554e939ae80d6bf47b36d5f199752d3c5eb98710c679566c9266128cb4a6a93ea85bd01390f448ee4b6ef4ca91b73b0d2c1b46368a6ef4736a9f86a2a91eb0b891f507b67459e53160983d28dc166b95734510d0c153d996631663afb7a524850a04a13755577d12c5a6607cb78c4c1223e77f633f81b2ac7e61d568208bddeaf562c434fcef085388a6318219948dd8de911283cef43fbab36c50e93bf6a3bf655a04a800d37f44b90f3d25dfb9cb69b01b339f38559c5bbce895a80679374d19fbfe0099b9e2809a98c916887d4b2c6f27bd4fcd71c9b39eb1093ac069c882f48c8f6c84cd3b294df4385b339d901d6db31cdb3aad69a190514fcdeb6e22b51f699558556253696a03d025831d20e6056c6b2df53c55c6c981f828b78f1e305d8cedd65b82c28082ae49a553ac9a5170ca37d163df61b41a4ce183d03a0b681690112a34949467d60de2d769921370aad149b40f296f6bd757a511b6b420d743d92d8ac61438791dc537a51612d4f8eb6c8b019ce89345c28d91e87881bf74aa6b2b69be91bba8f59f7e0818c01b0d339263200f9857d3499a0d8568937653440ec4c5c824423be1eb66825f3f884ce9d7eda53316aac736dba133a2304ef0e80f599db665619c863cefddc32f828b8d8064f9df7b11cbb3e0ed74d1e0c9d08aa7727b3c7187ebf2c91036e91532e872c8051ee4f6ceffa9bdfd5000ab3cf17e6f0e5217c9530475e062a40a2c020ceee5e25027c9ce00007424d4f6069769ef41f57d3f50d185a6e728fb1d0d4fbfc1a212facb4e407343e626f7a80840c171b3d5697b2b4d4ec123066687a82bdc4f7000000000000000000000000000000020c101b2129333c","attestation_type":"pqc_registry","signed_at":"2026-04-20 19:44:36"}],"hndl":null}