This page has been translated using TexTra by NICT. Please note that the translation may not be completely accurate.If you find any mistranslations, we appreciate your feedback on the "Request form for improving the automatic translation ".

Harunobu, a Machine-Readability and Rule-Compliance Checking Tool

In Digital Agency, in accordance with the "Rules for Machine-Readability in Administrative Data," a judgment tool called "Harunobu" is available on GitHub. This tool allows you to check and score how well your data complies with the rules. By checking the scoring results, you can understand which parts of the data should be corrected and improve the machine-readability.

Table of Contents

1. Harunobu

Harunobu is a judgment tool that checks the machine readability of Excel, CSV, and TSV files against the "Rules on Machine Readability in Administrative Data" and scores them out of 100 points. It was developed by Digital Agency and is available on GitHub. The results of the scoring show the locations of violations and the reasons for them, so you can consider areas for data improvement while viewing the results.

This tool can be operated in the user's local environment. It can also be incorporated into a system that processes data as a library. Unless the AI judgment support function is turned ON (* 1), judgments are made with mechanical logic in the program. Since the content of the loaded file is never sent to external servers, you can safely check unpublished administrative data and work files that are being edited and may contain confidential information.

  • * 1: Only if you optionally enable the "Decision-making assistance function based on AI," AI will be sent to APIs of external data services (Gemini, ChatGPT, Claude, etc.) set by you. When using the function, please make a decision in accordance with the security policy of your organization.

Harunobu's diagram illustrating the flow of rule conformance checking. The process is divided into five stages from left to right. The first stage is to upload the file. It is shown that the data to be inspected can be imported in file formats such as Excel, CSV, and TSV. The second stage is to perform a check. It is shown that the data can be graded in three levels according to the coded machine-readability rules. The third stage is to output the grading results. The locations of rule violations and the reasons can be confirmed, and the graded data is displayed as a diagram. For example, the problematic locations (cells in Kushiro, Muroran City) are highlighted in red. The fourth stage is data cleansing. The locations of rule violations and the reasons can be confirmed, and the graded data is displayed as a diagram. For example, the problematic locations (cells in Kushiro, Tokyo) are highlighted in red. The fourth stage is AI. The locations of rule violations and the reasons can be efficiently and mechanically corrected. The fifth stage is to ensure machine-readability. This can be used with AI and analytical tools. As an example of "data with improved machine-readability," a pre-formatted diagram with the column headings pref _ name and city _ name is displayed. Rule conformance check Harunobu can be used in the CLI, browser UI, and Python API, and can be used in stages 1 through 3.

2. What you can do (three forms of use)

Harunobu offers three types of usage according to the purpose of use. It can support a wide range of needs, from those in charge of practical work who want to check files intuitively by operating the screen, to engineers who want to automatically inspect a large amount of data and incorporate it into their own organization's systems. In either case, the judgment logic is the same, and you can download it from GitHub and run it safely in your local environment.

Mode of useOutlineExpected Usage Scenarios
Browser screen (Streamlit UI)Upload the file and check the grades and results on the screen.Verifying WebUI Operation
Command Line (CLI)Run on CLI (e.g. harunobu analyzedata. xlsx ) to get resultsRecurring Check Batch Processing
Call from Python APICall inside a program as a librarySystem Integration

3. How to use (browser screen)

In the browser screen (Streamlit UI), you can score in the following five steps. While watching the scored results, you can consider areas where data should be improved.

3-1. Upload Files

Select the file you wish to inspect. Excel, CSV, and TSV file formats are supported, and multiple files can be selected.

The file upload screen of "Harunobu, a machine readability check tool." On the left is the workflow navigation, with five steps arranged vertically: "1. File Upload," "2. Data Preview," "3. Layout Detection," "4. Run Check," and "5. Scoring Results." Currently, "1. File Upload" is displayed in red and active. In the main area on the right, the "File Upload" section shows buttons in the form xlsx / csv / tsv, indicating that harunobu _ sample. xlsx (22.4 kb) has already been uploaded, with the number of files: 1, format: XLSX, and total size: 0.02 MB. At the bottom of the screen is a button for "Next: Data Preview."

3-2. Preview Data

Check the loaded contents on the screen.

Data preview screen of "Machine Readability Check Tool Harunobu". In the left navigation, 1 is checked green and "2. Data Preview" is active (red). The uploaded file contains harunobu _ sample. xlsx (20.3 kb). There is a "Start over from the beginning" button at the bottom. In the right main area, the file name: harunobu _ sample. xlsx, Format: XLSX, Number of Sheets: 1 is displayed, and Sheet1 is being selected. The data of 15 rows, 8 columns, and 1 merged cell is displayed as a table, and the data of year / month, prefecture, municipality, item, and values 1 to 3 can be confirmed. The title line of "Machine Readability Test" and a multi-header including merged cells can also be confirmed. There is an "Upload" button on the left at the bottom of the screen, and a "Layout Detection" button on the right.

3-3. Detect Layout

Automatically estimates the layout of table ranges, headers, and more.

Layout detection validation screen of "machine-readability check tool Harunobu". In the left navigation, workflow 1 and 2 have green checks, and "3. Layout Detection" is active (red). The uploaded file is harunobu _ sample. xlsx (20.5 kb), and the scores by level are indicated as L1 (compulsory 0 points): 0, L2 (compulsory 0 points): 0, and L3: unchecked. At the bottom is a "Start over" button. Sheet1 is displayed in the "Select a sheet" column in the right main area. File name: harunobu _ sample. xlsx, number of detection tables: 1, and out-of-table cells: 2 are displayed. Sheet1 (15 rows x 8 columns) has tabs for original data and detection results, and a spreadsheet is displayed in the detection results tab. Labels such as "T1 Header [text]", "T1 Body [text]", "T1 Header [numeric]", "T1 Header [empty]", "T1 Body [mixed]", and "T1 Body [numeric]" are given in each cell. F5 cell has a note of "Cell merged multi-header".

3-4. Run Check

Select the level of check (lightweight check / standard check / detailed check) and execute judgment.

The screen for executing the check of "Harunobu, a machine readability check tool". In the left navigation, green checks are attached to workflows 1 to 3, and "4. Execute Check" is active (red). The uploaded file is harunobu _ sample. xlsx (20.5 kb), and the scores by level are indicated as L1 (mandatory 0 points): 0, L2 (mandatory 0 points): 0, and L3: unchecked. There is a "Start over from the beginning" button at the bottom. In the right main area, harunobu _ sample. xlsx is displayed as the target file. There are three radio buttons for selecting the level of check: "Lightweight Check," "Standard Check," and "Detailed Check," and standard check is selected. The items to be checked are "Level 1" and "Level 2," and "Structure + Data Quality Check" is described. "Data Preview" is on the left at the bottom of the screen, and the "Execute Check" button is on the right.

3-5. Check the scoring results and consider the points for improvement

The score for each level and a list of violated rules, locations of violations, and reasons are displayed. The results can be downloaded in JSON or CSV format, which can be used to investigate which parts should be corrected to improve machine readability. Levels with serious rule violations are given a "mandatory 0 point," and the reason for the violation is also clearly indicated.

The scoring results screen of "Harunobu, a machine-readability checking tool". In the left navigation, green checks are attached to the workflow from 1 to 4, and "5. Scoring Results" is active (red). The uploaded file is harunobu _ sample. xlsx (20.3 kb), and the scores by level are indicated as L1:65 points (pass 8/12), L2:0 points (fail 0 / pass 3/4), and L3: unchecked. At the bottom is a "Start over from the beginning" button. In the upper part of the right main area, as scores by level, there are three summary cards: Level 1:65 points (pass 8/12), Level 2:0 point (fail 0 / pass 3/4), and Level 3: unchecked. There is a red warning banner displayed at the bottom of the card: "Level 2 is a forced 0 point. This is due to a violation of the Serious rule (L2-02 (Fatal))." As a result of checking Sheet1 Table 1, the pass / fail of the Level 1 rule from L1-01 to L1-13 and the Level 2 rule from L2-01 to L2-06 is listed by card. Red indicates fail and gray indicates inapplicable. In the "Violation Locations" table at the bottom of the screen, 27 violations (error:4, info:7, warning:16) are listed with rule ID, sheet cell range, severity level, reason.

4. Usage (CLI)

Harunobu supports running from the command line (CLI) as well as the browser screen. Scoring can be performed with a single command, and the results can be output as JSON or CSV files, which can be used for routine checking of multiple files or for integration into existing workflows. For installation instructions and detailed usage of the commands, please check the README file at the GitHub Repository (GitHub) is located at Fork .

An image showing "An Example of Harunobu Running on the Command Line (CLI)". On the left, a sample image of an Excel spreadsheet is displayed as "Data to be inspected". It contains the following data: Year / Month, Prefecture, City / Ward, Item, and Values 1 to 3. Columns B to H, Lines 7 to 24. The range B7:H12 is highlighted with a red dashed frame. The cells F9:F10 are highlighted in red (data for January 2025:1,234 and 1,111). On the right is the "Scoring Results (CLI)" with the following text: "Range": "B7:H12", "Header Range": "B7:H7", "Rule ID": "L1-12", "Cell Range": "F9:F10", "Description": "Cell Merge (Data Area)" and "Rule ID": "L1-07", "Cell Range": "F9", "Description": "Multiple Data with Line Breaks".

5. Important Points

  • Harunobu is an implementation example based on "Rules for Machine Readability in Administrative Data". Please note that the judgment logic of Harunobu is not the rule itself. In addition, some rules are judged to be difficult to judge by machine, and are excluded from judgment in Harunobu.
  • Harunobu does not preclude the development of identification and correction programs based on the "Rules for Machine-Readability in Administrative Data." If you wish to develop programs that are your own evolution of Harunobu, the GitHub Repository (GitHub) is located at Fork in function.
  • For additional known limitations in Harunobu, please review the limitations. md (GitHub) file in the GitHub repository.

6. Inquiries

Any questions or ideas should come from , comments and requests, .

7. Related Information