Inclusion Criteria:

* Patients must be 5 to 60 years of age at the time of enrollment
* Patients with newly diagnosed untreated histologically confirmed classic Hodgkin lymphoma (cHL) (nodular sclerosis, mixed cellularity, lymphocyte-rich, or lymphocyte-depleted, or not otherwise specified \[NOS\]) with stage I or II disease
* Patients must have bidimensionally measurable disease (at least one lesion with longest diameter \>= 1.5 cm)
* Patients must have a whole body or limited whole body PET scan performed within 42 days prior to enrollment. PET-CT is strongly preferred. PET-MRI allowed if intravenous contrast enhanced CT is also obtained
* Pediatric patients (age 5-17 years) with known or suspected mediastinal disease must have an upright posteroanterior (PA) chest X-ray (CXR) for assessment of bulky mediastinal disease.

* Note: Pediatric patients who have received both a CT chest and upright PA CXR may meet the definition of bulk through either modality.
* Patients \>= 18 years must have a performance status corresponding to Zubrod scores of 0, 1 or 2
* Patients =\< 17 years of age must have a Lansky performance score of \>= 50
* Pediatric patients (age 5-17 years): A serum creatinine based on age/gender as follows (within 28 days prior to enrollment):

* 2 to \< 6 years (age): 0.8 mg/dL (male), 0.8 mg/dL (female)
* 6 to \< 10 years (age): 1 mg/dL (male), 1 mg/dL (female)
* 10 to \< 13 years (age): 1.2 mg/dL (male), 1.2 mg/dL (female)
* 13 to \< 16 years (age): 1.5 mg/dL (male), 1.4 mg/dL (female)
* \>= 16 years (age): 1.7 mg/dL (male), 1.4 mg/dL (female) OR a 24 hour urine creatinine clearance \>= 50 mL/min/1.73 m\^2 (within 28 days prior to enrollment) OR a glomerular filtration rate (GFR) \>= 50 mL/min/1.73 m\^2 (within 28 days prior to enrollment). GFR must be performed using direct measurement with a nuclear blood sampling method OR direct small molecule clearance method (iothalamate or other molecule per institutional standard)
* Note: Estimated GFR (eGFR) from serum or plasma creatinine, cystatin C or other estimates are not acceptable for determining eligibility
* For adult patients (age 18 years or older) (within 28 days prior to enrollment): Creatinine clearance \>= 30 mL/min, as estimated by the Cockcroft and Gault formula or a 24-hour urine collection. The creatinine value used in the calculation must have been obtained within 28 days prior to registration. Estimated creatinine clearance is based on actual body weight
* Total bilirubin =\< 2 x upper limit of normal (ULN) (within 28 days prior to enrollment)

* Unless due to Gilbert's disease, lymphomatous involvement of liver or vanishing bile duct syndrome
* Aspartate aminotransferase (AST) =\< 3 x ULN (within 28 days prior to enrollment)

* Unless due to Gilbert's disease, lymphomatous involvement of liver or vanishing bile duct syndrome
* Alanine aminotransferase (ALT) =\< 3 x ULN (within 28 days prior to enrollment)

* Unless due to Gilbert's disease, lymphomatous involvement of liver or vanishing bile duct syndrome
* Shortening fraction of \>= 27% by echocardiogram (ECHO), multigated acquisition scan (MUGA), or functional cardiac imaging scan (within 28 days prior to enrollment) or ejection fraction of \>= 50% by radionuclide angiogram, ECHO, MUGA, or cardiac imaging scan (within 28 days prior to enrollment)
* Diffusion capacity of the lung for carbon monoxide (DLCO) \>= 50% of predicted value as corrected for hemoglobin by pulmonary function test (PFT) (within 28 days prior to enrollment). If unable to obtain PFTs, the criterion is: a pulse oximetry reading of \> 92% on room air
* Known human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial
* For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated. Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load

Exclusion Criteria:

* Patients with nodular lymphocyte predominant Hodgkin lymphoma
* Patients with a history of active interstitial pneumonitis or interstitial lung disease
* Patients with a diagnosis of inherited or acquired immunodeficiency that is poorly controlled or requiring active medications, such as primary immunodeficiency syndromes or organ transplant recipients
* Patients with any known uncontrolled intercurrent illness that would jeopardize the patient's safety such as infection, autoimmune conditions, cardiac arrhythmias, angina pectoris, and gastrointestinal disorders affecting swallowing and/or absorption of pills
* Patients with a condition requiring systemic treatment with either corticosteroids (defined as equivalent to \> 10 mg daily predniSONE for patients \>= 18 years or \> 0.5 mg/kg \[up to 10 mg/day\] for patients \< 18 years) or other immunosuppressive medications within 14 days prior to enrollment

* Note: Replacement therapy such as thyroxine, insulin, or physiologic corticosteroid for adrenal or pituitary insufficiency is not considered a form of systemic treatment. Inhaled or topical steroids, and adrenal replacement doses (=\< 10 mg daily for patients \>= 18 years or =\< 0.5 mg/kg \[up to 10 mg/day\] predniSONE equivalents) are permitted in the absence of active autoimmune disease
* Note: Steroid use for the control of Hodgkin lymphoma symptoms is allowable, but must be discontinued by cycle 1, day 1
* Short term use of corticosteroids for premedication or treatment of an allergy or hypersensitivity is considered an acceptable use of corticosteroids.
* Patients with peripheral neuropathy \> grade 1 at the time of enrollment or patients with known Charcot-Marie-Tooth syndrome
* Patients with a prior or concurrent malignancy whose natural history or treatment has the potential to interfere with the safety or efficacy assessment of the investigational regimen
* Administration of prior chemotherapy, radiation, or antibody-based treatment for cHL
* Prior solid organ transplant
* Prior allogeneic stem cell transplantation
* Live vaccine within 30 days prior to planned day 1 of protocol therapy (e.g., measles, mumps, rubella, varicella, yellow fever, rabies, bacillus Calmette Guerin \[BCG\], oral polio vaccine, and oral typhoid). Administration of messenger ribonucleic acid (mRNA) vaccines are permitted
* Female patients who are pregnant since fetal toxicities and teratogenic effects have been noted for several of the study drugs. A pregnancy test within 28 days prior to enrollment is required for female patients of childbearing potential
* Lactating females who plan to breastfeed their infants starting with the first dose of study therapy and for at least 6 months after the last treatment
* Sexually active patients of reproductive potential who have not agreed to use a highly effective contraceptive method for the duration of their study drug therapy. Following therapy, patients will be advised to use contraception as per institutional practice or as listed below for investigational agents, whichever is longer

* Men and women of childbearing potential must continue contraception for a period of 6 months after last dose of brentuximab vedotin
* Women of child-bearing potential (WOCBP) must continue contraception for a period of at least 5 months after the last dose of nivolumab
* All patients and/or their parents or legal guardians must sign a written informed consent
* All institutional, Food and Drug Administration (FDA), and National Cancer Institute (NCI) requirements for human studies must be met