How to Use Google AI Studio as a Screen Assistant

TL;DR

Google AI Studio can watch a shared screen, listen through a microphone, and provide spoken or written guidance while you work in applications such as QuickBooks and Excel. To use it, open Stream Real Time, choose audio or text output, select a voice if needed, enable the microphone, share a window or the entire screen, and ask direct questions.

Transcript

there's an AI app from Google that is now my favorite AI app of all time and I use it even more than chat GPT and I know that is a very bold claim but I think once you watch this quick demo you'll agree with me and then I'll show you exactly how to use it in more detail with more examples this video is sponsored by HubSpot more on that in a bit hey... Read More

Key Insights

• Google AI Studio is presented as a free tool that can observe a shared screen and provide conversational guidance while someone works. Its real-time interaction allows users to ask questions about visible software, documents, forms, and tasks without manually describing every on-screen detail.
• Stream Real Time works by combining screen sharing, microphone input, and Gemini responses. The user can share a specific window or an entire screen, allowing the assistant to examine different tabs and applications as the user moves between them.
• Audio output creates a spoken assistance experience with minimal perceived delay. Google AI Studio also displays responses as text, and users can choose text-only output when spoken guidance is unnecessary or less convenient for their working environment.
• The QuickBooks demonstration shows that screen-aware AI can compare information across an expense spreadsheet and a transaction list. It identifies tips costing $40 and parking costing $60 as missing, then reports a total expense cost of $857.
• Receipt recognition allows the assistant to extract visible information from a document shown on screen. In the example, it identifies a parking receipt from Chicago Midway Airport, a total cost of $60, and the dates displayed on the receipt.
• System instructions can shape how the assistant communicates during a session. The presenter enters a request for short and direct responses before starting the Excel demonstration, showing that response style can be configured without repeating the preference in every question.
• The Excel demonstration establishes that a pivot table is a data summarization tool used to reorganize and analyze spreadsheet information. The assistant guides the user to select the data, open the Insert tab, choose PivotTable, confirm the range, and create it on a new worksheet.
• Typed prompts provide an alternative input method when the user does not want to speak. The assistant can still use the shared screen as context, so users can combine visible information with written instructions and familiar prompting techniques used with other chatbots.

Questions & Answers

Q: How do you use Google AI Studio as a screen assistant?

Sign in to Google AI Studio with a Google account, open Stream Real Time, and choose an output format. Audio provides spoken responses, while text is also available. Enable the microphone, select the screen-sharing option, and share either a specific window or the entire screen. Once the stream begins, ask conversational questions about whatever is visible on the shared screen.

Q: What can Google AI Studio see during a real-time stream?

Google AI Studio can see the content that the user chooses to share. On a Mac, the demonstration allows selection of a specific window or the entire screen. Sharing the entire screen lets the assistant observe different tabs and applications as the user works. The interface also offers camera access, and users can share the screen, the camera, or both.

Q: How can Google AI Studio help with QuickBooks expenses?

Google AI Studio can examine visible QuickBooks transactions alongside an expense report and help identify items that do not appear in both places. In the demonstration, it finds missing tips of $40 and parking of $60. It also explains how to open the expense form, interpret fields such as category and tags, enter the amount, and save the transaction.

Q: Can Google AI Studio read and interpret a receipt?

Google AI Studio can interpret receipt information that is visible through the shared screen or camera. In the example, it identifies the document as a parking receipt for Chicago Midway Airport, reads a total cost of $60, and reports the dates displayed on it. This allows a user to ask about a receipt while continuing an expense-entry workflow.

Q: How do you enable spoken answers in Google AI Studio?

Open Stream Real Time and set the output format to audio instead of text. The interface also lets the user choose a voice, with Puck used during the demonstration. Next, activate the microphone and begin sharing the relevant screen or window. The assistant can then listen to spoken questions and answer aloud while also providing text in the interface.

Q: How can system instructions change Gemini's responses?

System instructions let the user specify a preferred tone or response style before beginning the stream. In the Excel example, the instruction asks for short and direct responses. The field can also be left blank, as it was during the earlier demonstration. This setting helps maintain a communication style without adding the same preference to every spoken or typed prompt.

Q: What is a pivot table, and how do you create one in Excel?

A pivot table is a data summarization tool that reorganizes and analyzes spreadsheet data to reveal patterns or insights. To create one, select the relevant data and headers, or select a cell within the data range. Open the Insert tab, choose PivotTable, confirm that the displayed range includes the intended data, select a new worksheet if desired, and press OK.

Q: Can you type prompts instead of speaking in Google AI Studio?

Google AI Studio accepts typed prompts when the user does not want to have a spoken conversation. The prompt field works similarly to the input area in other chatbots, but the shared screen supplies additional context. Users can therefore type questions about visible applications, combine those questions with established prompting techniques, and receive responses in the selected text or audio format.

Summary & Key Takeaways

• Google AI Studio’s Stream Real Time feature lets Gemini observe a shared screen and respond through audio or text. The demonstration shows it comparing an expense report with QuickBooks, identifying missing expenses, reading a parking receipt, explaining form fields, and calculating the expense total while the user works.

• The setup begins by signing in with a Google account and opening Stream Real Time. Users can select audio as the output format, choose a voice such as Puck, activate the microphone, and share a specific window or the entire screen. Camera access is also available but is not required.

• A second demonstration shows the assistant guiding a user through Excel. It identifies a sales order spreadsheet, points out the zoom controls, defines a pivot table, and explains how to create one. System instructions can request short, direct responses, while typed prompts provide an alternative to spoken conversation.