Inclusion Criteria:

* REGISTRATION TO STEP 1
* Patient must have histologically confirmed HER2 negative metastatic esophageal or gastric adenocarcinoma (American Joint Committee on Cancer \[AJCC\] 8th edition)
* Patient must have oligometastatic disease at the time of registration, which is defined as the following:

* At most 3 radiologically visible metastatic lesions (not sites), in addition to the primary site. Computed tomography (CT) or magnetic resonance imaging (MRI) scans will be performed for staging purposes. Patients with oligometastatic sites that are only detected with positron emission tomography (PET)/CT will be eligible for participation, as long as radiation planning and administration is feasible after discussion with treating radiation oncologist. Malignant lymph node should be at least 1 cm in size or biopsy proven involved by disease
* Anatomically defined lymphadenopathy will be considered as 1 site of metastatic disease. For example, 2 enlarged paraaortic lymph nodes will be considered as one site, and 2 additional sites will be allowed to meet protocol definition of oligometastatic disease. However, if supraclavicular or cervical nodes are involved for distal esophageal tumors or gastric tumors, these are counted separately from intrathoracic nodes. For upper thoracic/cervical esophageal tumors, the involvement of celiac nodes are counted separately from intrathoracic nodes. Intrathoracic nodes, defined as hilar and mediastinal nodes, will be collectively counted as one
* Patients with radiologically evident peritoneal metastasis will be excluded.
* Patient must have an Eastern Cooperative Oncology Group (ECOG) performance status 0-1
* Women of childbearing potential and sexually active males must not expect to conceive or father children by using accepted and effective method(s) of contraception (both double barrier contraception and birth control pills or implants) or by abstaining from sexual intercourse for at least one month after the last dose of protocol treatment and continuing for 5 months after the last dose of protocol treatment (for female patients) and for 7 months after the last dose of protocol treatment (for male patients who are sexually active with women of child bearing potential \[WOCBP\]). Investigators must counsel WOCBP and male patients who are sexually active with WOCBP on the importance of pregnancy prevention and the implications of an unexpected pregnancy
* Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (obtained within 28 days prior to registration)
* Hemoglobin \>= 8 g/dL (obtained within 28 days prior to registration)
* Platelets (PLT) \>= 100 x 10\^9/L (obtained within 28 days prior to registration)
* Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 3.0 x upper limit of normal (ULN) (obtained within 28 days prior to registration)
* Bilirubin =\< 1.5 x institutional ULN (obtained within 28 days prior to registration)
* Serum creatinine =\< 1.5 x institutional ULN (Cockcroft and Gault formula) (obtained within 28 days prior to registration)
* Albumin \> 2.5 g/dL (obtained within 28 days prior to registration)
* Patient must be able to understand and willing to sign and date the written voluntary informed consent form prior to any protocol-specific procedures
* Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial. Patients must have CD4 \> 200 at the time of registration

* NOTE: HIV testing is not required for eligibility
* Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial
* Patients who had prior definitive treatment for early stage EGA with either surgery or chemoradiation are eligible for participation as long as recurrent disease developed at least 6 months after completion of all prior therapies
* Any major surgery must have been completed \>= 4 weeks prior to registration
* REGISTRATION TO STEP 2
* Patient must have histologically confirmed HER2 negative metastatic esophageal or gastric adenocarcinoma (AJCC 8th edition) with stable disease after about 4 months of fluorouracil, leucovorin calcium, and oxaliplatin (FOLFOX) or 6 cycles of capecitabine and oxaliplatin (CAPOX) (Step 1 treatment)
* Patient must have no evidence of disease progression based on Response Evaluation Criteria in Solid Tumors (RECIST) criteria since Step 1 registration. Patients with complete radiologic response are eligible for Step 2
* Patient must have an ECOG performance status 0-1

Exclusion Criteria:

* Patient must not have any contraindications to 5-fluorouracil (5-FU) or capecitabine, oxaliplatin
* Patient must not have any contraindications to radiation therapy based on consultation with a radiation oncologist. Formal radiation oncology evaluation will be required for eligibility purposes. Prior palliative or definitive radiation to the primary site is allowed, as long as it was completed at least 2 weeks before registration
* Women must not be pregnant or breast-feeding due to the potential harm to unborn fetus and possible risk for adverse events in nursing infants with the treatment regimens being used

* All females of child bearing potential must have a serum or urine pregnancy test to rule out pregnancy within 14 days prior to registration
* A female of childbearing potential is any woman, regardless of sexual orientation or whether they have undergone tubal ligation, who meets the following criteria: has achieved menarche at some point, has not undergone a hysterectomy or bilateral oophorectomy; or has not been naturally postmenopausal (amenorrhea following cancer therapy does not rule out childbearing potential) for at least 24 consecutive months (i.e., has had menses at any time in the preceding 24 consecutive months)
* Patient must not have had any prior treatment with 5-FU or capecitabine and/or oxaliplatin containing systemic therapy

* NOTE: Patients previously treated with radiosensitizing doses of 5-FU will be eligible for participation as long as adequate time has elapsed from past treatments
* NOTE: Patients who received systemic 5-FU or capecitabine and/or oxaliplatin as part of the treatment for their locoregional disease are eligible for participation, as long as all definitive therapy has been completed at least 6 months prior to trial enrollment
* Patients with known central nervous system (CNS) metastasis will be excluded from trial participation, regardless of the status of the CNS disease
* Patient must not have any uncontrolled intercurrent illness including, but not limited to ongoing or active infection requiring treatment, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
* Patient must not have had live vaccines within 30 days prior to registration. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, chicken pox, yellow fever, rabies, Bacillus Calmette Guerin (BCG), and typhoid (oral) vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g., Flu-Mist) are live attenuated vaccines and are not allowed