In a large skillet or wok, heat the olive oil over medium-high heat. Add garlic and ginger; stir-fry until fragrant, about 30 seconds.
Add tofu; stir-fry until golden brown, about 2 minutes. Add red and yellow peppers; cook, stirring, until peppers are crisp-tender, about 3 minutes.
Add rice and soy sauce; cook, stirring occasionally, until rice is heated through, about 3 minutes. Stir in sesame oil. Season with salt and pepper; garnish with scallions. Serve hot.