The academic document: introduction, theoretical framework, product, CRISP-DM methodology, tables and reproducibility.
Introduction
Background on the statutory regime, operational definition of the problem, objectives, justification along a triple contributory axis, research questions, and empirically verifiable qualitative hypotheses.
1.1 Background
The Immigration and Nationality Act (INA) of 1965 [10] establishes the current system of annual quotas by category with a statutory 7 % limit per country (per-country limit), which is the primary source of the severe retrogression observed in high-demand countries such as Mexico, India, China, and the Philippines [6].
The Department of State publishes the Visa Bulletin [7] monthly with two calendars: Final Action Dates (FAD, published since 1992; the pipeline's homogeneous monthly series spans Dec-2001–present, 296 obs), which authorizes final adjudication, and Dates for Filing (DFF, since October 2015 with 130 obs), which authorizes the early start of the adjustment process. Bier [60] documents that the employment-based categories of countries under an effective limit, particularly India, have experienced retrogressions of several years in recent periods.
Prior work: Vegesana [11] applies discriminative classifiers to approval forecasting; Jain et al. [12] propose an ARIMA-LSTM hybrid for series with complex dynamics; Carammia et al. [13] and Pu et al. [9] integrate machine learning with data at scale in migration flows.
1.2 Problem definition
No published academic work was identified, in the preliminary review, offering an open, reproducible, and systematically evaluated predictive system over the Visa Bulletin multiseries panel with 95 % prediction intervals. Existing commercial platforms are black boxes with no methodology, data, or evaluation protocol published in an auditable way.
The direct consequence is that millions of applicants must plan medium- and long-term decisions (family, professional, and financial stability) on opaque estimates or low-quality methods. Backlog figures are documented in Section 1.1 [4], [6].
The need is to build a public longitudinal database of the Visa Bulletin 1992–2026 and an evaluable predictive system that turns more than three decades of data into auditable forecasts with 95 % prediction intervals, filling a verifiable gap in the applied literature.
1.3 Objectives
General objective
Develop and implement an applied predictive system for Visa Bulletin priority dates, organized as a multiseries panel indexed by chargeability country, immigrant category, and table type, with monthly forecasts at horizons of 1, 3, 6, and 12 months accompanied by 95 % prediction intervals, under the CRISP-DM methodology [64].
Specific objectives
- Build the longitudinal database of the Visa Bulletin 1992–2026 structured by country × category × table × month, published under an open license.
- Empirically characterize the historical behavior of priority dates and their administrative regimes by chargeability country or area.
- Design and implement the full family of models listed in Table 3, without privileging any architecture in advance.
- Evaluate predictive performance through expanding walk-forward temporal validation with scaled metrics (sMAPE, MASE, MAE, RMSE) and 95 % prediction intervals, reporting aggregate and disaggregated results by country, category, and table type.
- Deliver an academic demonstration application that allows querying historical series, forecasts, and 95 % prediction intervals with explicit warnings about the informational and non-legal nature of the estimates.
1.4 Justification
Social contribution
The process of obtaining permanent residence affects millions of people in terms of family, work, and financial stability. Nearly 4 million remain abroad awaiting a family-based visa [6], within the global USCIS backlog of 11.5 million pending cases across all forms [4]. The project supports applicants' planning with 95 % prediction intervals and greater transparency than the closed, non-auditable tools on the market.
Technical contribution
The system addresses observable limitations of closed commercial platforms, particularly the lack of transparency regarding data, methodology, and evaluation protocol. The integration of a complete multiseries panel (multi-country × multi-category × multi-table) under a rigorous comparative framework constitutes an applied contribution to the forecasting of migratory phenomena.
Academic contribution
- Open database 1992–2026 with a permissive license, pre-registered evaluability criteria, and verifiable reproducibility (R1–R7, Appendix A.3).
- Named comparative framework under CRISP-DM with a family of linear and nonlinear models evaluated without privileging architectures in advance.
- Uncertainty quantification with three mechanisms for the 95 % prediction intervals: analytical ARIMA, MC dropout [40], and conformal prediction [59].
1.5 Questions and hypotheses
Consistent with the professionalizing profile of the MIAAD program, the questions are framed as open questions and the hypotheses as soft qualitative statements that will be tested empirically against the pilot data. Formal statistical rigor (Diebold-Mariano + Holm correction at ) is documented as a methodological decision in Chapter IV, not as a pre-committed threshold in the body.
General question
With what accuracy and with what calibration of the 95 % prediction intervals can the future behavior of the Visa Bulletin multiseries panel be forecast under a CRISP-DM comparative framework, over the coverage of evaluable series and respecting the structural heterogeneity among country–category–table cells?
Specific questions
- Patterns. What patterns of advancement, stagnation, and retrogression do the series present by chargeability country and immigrant category, and what proportion of each series is evaluable under the pre-registered criteria?
- Predictive performance. Which of the implemented models —linear (ARIMA, SARIMA, Prophet) or nonlinear (LSTM, ARIMA-LSTM, DeepAR, XGBoost)— offers the best predictive performance in sMAPE and MASE over the evaluable pilot set, and with what consistency across strata?
- Series characteristics. What structural characteristics (effective length, frequency of retrogressions, frequency of Current/Unavailable states) explain predictive difficulty per cell?
- Coverage. What empirical coverage do the 95 % prediction intervals achieve relative to their nominal level, in aggregate and by stratum?
Hypotheses
The best nonlinear model will improve on the strongest linear one in a material proportion of the pilot series.
The best nonlinear model in the evaluated family {LSTM, ARIMA-LSTM, DeepAR, XGBoost} will reduce the central metrics (sMAPE, MASE) relative to the strongest linear model (ARIMA or Prophet, depending on the stratum) in a material proportion of the evaluable pilot series. The identity of the winning model is an empirical question that the experiment will answer with data.
Predictive difficulty varies by stratum and the intervals may deviate in unstable regimes.
Predictive difficulty —measured as MASE under the empirical winner— will show an association with the effective length of the series and with the frequency of discontinuities (retrogressions, C/U transitions). The 95 % prediction intervals will achieve empirical coverage close to nominal in the aggregate, but may deviate in strata with an unstable administrative regime.
These hypotheses are interpreted in terms of predictive associations that are empirically verifiable with the project's data, not as causal claims about the administrative dynamics of the Department of State. Causal interpretation is explicitly out of scope.
Theoretical and technological framework
Eight theoretical-framework subsections (§2.1.1–§2.1.8) range from the U.S. immigration system to the heterogeneity of multiseries forecasting, plus five technological-framework subsections (§2.2.1–§2.2.5). Each theoretical subsection closes with an Implication for this project paragraph that connects the content to concrete design decisions.
Immigration system & Visa Bulletin
INA 1965 [10], annual quotas, the 7 % per-country limit, Final Action Dates (FAD) and Dates for Filing (DFF) [7] calendars. Convention: F1–F4 = {F1, F2A, F2B, F3, F4} (five series per chargeability country), not an exclusive range.
Classical time series
Classical decomposition . Box & Jenkins [24], [29]; Hamilton [25]; Hyndman & Athanasopoulos [26]. Stationarity tests ADF [27] and KPSS [28] (with opposite null hypotheses), AIC [30] and BIC [31] criteria. ARIMA, SARIMA, and Prophet [32] as linear reference models.
DL fundamentals for series
Backpropagation [36], MLP, regularization (weight decay, dropout [39], recurrent dropout [40], batch normalization [41]) and early stopping. ReLU [37] and the Adam optimizer [38] as standards. Components used in the nonlinear models of the comparative framework.
RNN and LSTM
Elman [42], Bengio et al. [37]: the vanishing gradient problem. Hochreiter & Schmidhuber [43] introduce the LSTM with memory cells and gates. BiLSTM [44], [45] and GRU [47] variants. The LSTM constitutes the nonlinear component of the ARIMA-LSTM hybrid.
Hybrids and modern architectures
Zhang [19] formalizes linear+nonlinear hybridization; ARIMA-LSTM in epidemiology [12], [21], economics [51], energy [52], and trade [20]. The M4 Competition [50] and ES-RNN [49] show that combinations dominate. DeepAR [22] (global) and XGBoost [56] (tabular) complete the compared family; TFT [54], N-BEATS [53], and PatchTST [55] only as reference.
Learning applied to migratory phenomena
Vegesana [11] applies discriminative classifiers to approval forecasting; Carammia et al. [13] forecast asylum flows with machine learning and large-scale data; Pu et al. [9] review methods and sources; Hoffmann Pham & Luengo-Oroz [57] on predictive modeling of refugee movements.
Metrics, validation, and 95 % intervals
Hyndman & Koehler [16]: MASE as a universal metric scaled by the seasonal naïve. Walk-forward from Tashman [58] and Bergmeir & Benítez [17]. Diebold-Mariano [18] with Holm correction for the pre-registered family. 95 % prediction intervals via three mechanisms: analytical ARIMA, MC dropout [40], and conformal prediction [59].
Multiseries forecasting and heterogeneity
Local models (trained per cell), global models (DeepAR [22] over the panel), and tabular models (XGBoost [56]). The choice among regimes is an empirical question that the project answers with data, given the small-data regime per series (296 obs FAD, 130 obs DFF).
Language and libraries: Python 3.x with pandas, numpy, statsmodels, scikit-learn [62], XGBoost [56], PyTorch [63], Prophet [32]. Visualization: matplotlib, seaborn. Version control: Git/GitHub under an open license. Documentation: LaTeX/Overleaf. Reproducibility: version manifests (requirements.txt), recorded seeds, MLflow for hyperparameter logging.
Expected product and validation
Description of the product, scope governance, and form of validation. The chapter responds strictly to «what is delivered?» and «how is it validated?»; the operational decisions (parameters, hyperparameters, testing scheme) are documented in Chapter IV.
3.1 Description of the solution
3.1.1 · Methodological framework
A comparative framework that does not privilege architectures in advance, instrumented under CRISP-DM [64]. Three complementary families: (a) linear models (seasonal naïve, ARIMA, SARIMA, Prophet), (b) local nonlinear models (LSTM, ARIMA-LSTM), (c) global and tabular models (DeepAR, XGBoost). The identity of the best-performing model is an empirical question of the study.
3.1.2 · Tentative architecture
Note (July 2026): the figure is the literal tentative architecture from the May 2026 proposal, whose comparison framework proposed that initial set of candidate models. The framework actually executed grew during development — the current count is derived from the governed model catalog and is reported in the prospective evaluation.
3.1.3 · Analytical coverage at three levels
The system's coverage is organized into three explicit levels:
- Structural database. Contains all combinations of country (Mexico, India, China, Philippines, All Chargeability Areas Except Those Listed) × category (family-based F1, F2A, F2B, F3, F4 and employment-based EB-1 through EB-5 with subcategories) × table (FAD, DFF).
- Evaluable coverage. Series with sufficient history, continuity, and variability for forecasting, according to the pre-registered criteria in Table 2.
- Initial pilot coverage. Family-based categories F1–F4 over the high-demand countries and the residual grouping, starting with Mexico because of the severity of its backlogs.
3.1.4 · Predicted variable
Let be the priority date transformed into days since a base date, for the cell (country , category , table ) in month . The variable is continuous and constitutes the target of a single temporal regressor. The system is trained exclusively on observations with state (specific date); Current and Unavailable observations are preserved in the database as descriptive annotation but are not a predictive target. Retrogressions within the regime are kept as legitimate observations (a genuine phenomenon of the immigration system).
3.1.5 · Training isolation
The regression metrics are computed exclusively on observations with a specific date. This avoids contaminating the evaluation with categorical observations (Current/Unavailable) that fall outside the system's predictive target. The descriptive characterization of the C/U/F regimes lives in the exploration report (Ch. IV, Phase 2), not in the operational metrics.
3.2 Delimitations, limitations, and governance
The scope is bounded through an explicit set of delimitations (sovereign decisions of the researcher), while the constraints imposed by external factors are acknowledged as limitations.
Researcher decisions
- Geography: MX · IN · CN · PH · All Charg.
- Tables: FAD & DFF (evaluated separately)
- Horizons: h = 1, 3, 6, 12 months
- Models: 8 compared (linear + nonlinear)
- Metrics: sMAPE · MASE · MAE · RMSE
- Validation: expanding walk-forward
- Application: academic demonstration from a repository
External factors
- Data: 296 obs FAD, 130 obs DFF per series
- Structural discontinuities in the regulatory framework
- Variable composition of All Chargeability
- Academic computational resources (not production-grade)
- Regulatory volatility with no observed impact in the period (e.g., recent policy announcements [3])
Scope governance
(i) Reproducible longitudinal CSV database under an open license; (ii) protocol of evaluable series (Table 2); (iii) implementation of the complete family (Table 3) under a single temporal regressor trained on observations over the pilot coverage; (iv) walk-forward with regression metrics and calibration of the 95 % prediction intervals.
Academic demonstration application runnable from the repository, with version manifests. There is no commitment to continuous public deployment: the system is academic, evaluated and reproducible.
3.3 Form of validation
Chapter III describes what is validated and at what level success is reported. The operational decisions (walk-forward parameters, confirmatory testing scheme, family-wise error control) are documented in Chapter IV.
Expanding walk-forward
Temporal validation with a training set that advances one month at a time, respecting causality. Reporting by horizon ( months).
Scaled metrics
sMAPE, MASE, MAE, and RMSE formally defined in §2.1.7. MAPE only as a complement. Calibration: empirical coverage of the 95 % prediction intervals vs. nominal.
Statistical comparison
Formal comparison between the best nonlinear model and the strongest linear one over the pre-registered family of pilot cells (specific instrumentation in Ch. IV §4.4).
95 % prediction intervals
Three complementary mechanisms: analytic ARIMA under Gaussian assumptions, MC dropout [40] for neural networks, and conformal prediction [59] as a model-agnostic reference.
Qualitative validation
Case studies on critical cells and bounded retrospective validation over bulletins published after the training cutoff; a readable interpretation for end users with a disclaimer.
Threats to validity
Internal validity (data leakage), external (generalization), construct (proxies of difficulty), and statistical (sample size, preliminary power).
Success levels
Honest report of the experiment
Reproducible construction of the dataset, complete execution of the walk-forward over the pilot coverage, reporting of scaled metrics and 95 % prediction intervals. It does not require any nonlinear model to outperform the linear ones: reporting a null result also satisfies the minimum level.
Material and verifiable improvement
The best nonlinear model in the family {LSTM, ARIMA-LSTM, DeepAR, XGBoost} improves on the strongest linear ones in a material proportion of the pilot series, with a verifiable magnitude in sMAPE/MASE. The specific statistical instrumentation lives in Ch. IV §4.4.
Calibration + stable log
The 95 % prediction intervals achieve empirical coverage close to nominal in the aggregate. Retrospective log over at least 6 monthly bulletin cycles following the training cutoff. Functional demonstration application with a disclaimer.
CRISP-DM methodology
The project is implemented under CRISP-DM (Cross-Industry Standard Process for Data Mining) [64], an established methodology since 2000 that has become the de facto standard for data mining and applied machine learning projects. Its six canonical phases map to the project's five operational phases.
Business & data understanding
Addressed in Chapters I and II of the proposal. It defines the problem, the affected stakeholders (millions of applicants), the backlog figures, the INA 1965 statutory regime, the structure of the Visa Bulletin, and the high-level methodological decisions.
Exploratory analysis
Characterization of the historical behavior per evaluable series: STL decomposition, ADF [27] and KPSS [28] tests, quantitative identification of retrogressions and C/U periods, application of the exclusion criteria from Table 2.
Preparation & modeling
Construction of the multiseries panel and training of the 8 models proposed in Table 3: linear (naïve, ARIMA, SARIMA, Prophet), nonlinear local (LSTM, ARIMA-LSTM), and global/tabular (DeepAR, XGBoost). Hyperparameter selection without data leakage; generation of 95 % prediction intervals via three mechanisms.
Evaluation
Expanding walk-forward with scaled metrics (sMAPE, MASE, MAE, RMSE) by horizon and aggregated. Computation of empirical coverage of the 95 % intervals. Formal statistical comparison with Diebold-Mariano [18] over the pre-registered family of comparisons, with Holm correction at familywise . Diagnosis of overfitting and leakage, threats to validity.
Deployment
Publication of the dataset under an open license, freezing of the repository (R1–R7), an academic demonstration application with usage warnings (the disclaimer in Appendix A.5), and a comprehensive final report. There is no commitment to a permanent public service: the system is academic, evaluated, and reproducible, not operational.
The five phases are distributed over 10 months with weekly granularity (40 active weeks + the December 2026 academic recess), aligned with the MIAAD program's PI-I (Aug–Nov 2026) and PI-II (Jan–May 2027) periods. Verifiable milestones at each phase close, with review of the deliverable document and assessment by the thesis director.
Key visualizations
Proposed coverage, methodological exclusion criteria, compared baselines, and a country×category×table matrix with three coverage levels (structural / evaluable / pilot).
Table 1 · Proposed analytical coverage
| Dimension | Admitted values | Operational notes |
|---|---|---|
| Country / chargeability | Mexico, India, China, Philippines, All Chargeability Areas Except Those Listed | The last one is not a country but an administrative grouping with variable composition. |
| Family categories | F1, F2A, F2B, F3, F4 | Convention F1–F4 = the full set {F1, F2A, F2B, F3, F4}. |
| Employment categories | EB-1, EB-2, EB-3, EB-4, EB-5 (+ subcategories) | Schematic; includes Other Workers, Certain Religious Workers, set-asides. |
| Table type | Final Action Dates (FAD), Dates for Filing (DFF) | Evaluated separately, with no direct cross comparison. |
| Special codes | Current (C) and Unavailable (U) | Kept as descriptive annotation in the database; they do not participate in the regressor's training (§3.1.5). |
Structure of the multiseries panel . Source: own elaboration.
Table 2 · Methodological exclusion criteria
| Criterion | Operational description |
|---|---|
| Insufficient length | Fewer than 60 monthly observations after filtering missing data; prevents building a viable walk-forward partition. |
| Zero variability | Series with a constant value (e.g., permanent Current) over the evaluated period; provides no predictive information. |
| Non-recoverable missing data | Three or more consecutive months unpublished with no possible recovery from DOS archives. |
| Non-harmonizable regulatory changes | Administrative modifications that alter the category's definition (e.g., redefinition of EB subcategories). |
| Absence of a specific date | The combination never published a specific date; only Current or Unavailable. No numerical target to train the regressor. |
| Variable composition | Applicable to All Chargeability Areas Except Those Listed: the composition of countries changes over the period (compositional non-stationarity). It will be accompanied by an explicit warning in any inference. |
Excluded series are reported descriptively but do not enter the quantitative pipeline. Source: own elaboration.
Table 3 · Compared models (8 candidates of the comparative framework)
| Model | Type | sMAPE target | MASE target | Role in the comparative framework |
|---|---|---|---|---|
| Seasonal naïve | Naive baseline | — | 1.00 | Minimum reference. |
| ARIMA | Univariate linear [24], [29] | < 25 % | < 0.97 | Linear model without seasonality. |
| SARIMA | Seasonal linear | < 22 % | < 0.94 | Captures annual fiscal-year cycles. |
| Prophet | Additive with changepoints [32] | < 20 % | < 0.92 | Models retrogressions as regime changes. |
| Pure LSTM | Univariate deep [43] | < 20 % | < 0.90 | Nonlinear capacity without a linear component. |
| ARIMA-LSTM | Local hybrid [12], [19] | < 17 % | < 0.88 | Hybrid linear + nonlinear reference per series. |
| DeepAR | Global multiseries deep [22] | < 17 % | < 0.88 | Learns transfer across panel series. |
| XGBoost + regressors | Tabular with exogenous variables [56] | < 18 % | < 0.90 | Alternative with calendar features. |
No model is privileged beforehand as «central»: the comparative framework will empirically determine which one(s) lead in each stratum. Indicative target thresholds. Source: own elaboration.
Figure · Country × category × table coverage matrix
| Country / Category | F1 | F2A | F2B | F3 | F4 | EB-1 | EB-2 | EB-3 | EB-4 | EB-5 |
|---|---|---|---|---|---|---|---|---|---|---|
| Mexico 🇲🇽 | ★ | ★ | ★ | ★ | ★ | ● | ● | ● | ○ | ○ |
| India 🇮🇳 | ★ | ★ | ★ | ★ | ★ | ● | ● | ● | ○ | ○ |
| China 🇨🇳 | ★ | ★ | ★ | ★ | ★ | ● | ● | ● | ○ | ● |
| Philippines 🇵🇭 | ★ | ★ | ★ | ★ | ★ | ● | ● | ● | ○ | ○ |
| All Charg. * | ● | ● | ● | ● | ● | ● | ● | ● | ○ | ○ |
FAD published since 1992; homogeneous pipeline series Dec-2001–today (296 obs). DFF since October 2015 (130 obs). Each pilot cell is reported aggregated and disaggregated. Source: own elaboration.
Reproducibility annex
In line with the commitment stated in Section 3.2, the public repository includes the following seven components verifiable by third parties. Obtaining the academic DOI is conditional, not a commitment of the minimum deliverable.
If the author decides to request it, archived in Zenodo or equivalent, with a persistent unique identifier linked to the commit hash frozen on the defense date. Publication under an open license (MIT or equivalent) is indeed a firm commitment.
requirements.txt with pinned versions (==), optionally complemented with environment.yml for Conda.
Each experiment documents the seed(s) used for NumPy, PyTorch, and Scikit-learn at the start of each notebook or script.
Memory, CPU model, presence/absence of a GPU, operating system, and architecture (Apple Silicon, x86_64) in REPRODUCIBILITY.md.
Times per pipeline stage (ingestion, features, training, walk-forward, evaluation) reported as indicative information.
SHA-256 checksums published for raw (data/raw/) and processed (data/processed/) data; bit-by-bit verification.
Complete record of explored configurations and finally selected configurations per model and per stratum (MLflow or equivalent).
src/ · data/raw/ · data/processed/ · notebooks/ · models/ · tests/ · docs/ · README.md with step-by-step instructions to reproduce every figure and table in the manuscript from scratch, together with REPRODUCIBILITY.md consolidating R1–R7 and docs/disclaimer_academico.txt, which is automatically injected into every output of the academic demonstrator.
Note (July 2026): this listing is the literal commitment from the May 2026 research proposal. The structure actually delivered evolved — two layered packages instead of a single src/, zero notebooks, per-profile locks instead of requirements.txt — and is documented as-is in the Structure section.