Flow Reader: Empowering Those with Reading Difficulties through Assistive Technology

Pausing the Project: More Useful to Me than Market Alternatives

I've paused this side project as it became too much of a time commitment. I developed the application to a point where it is personally more useful to me than other solutions available on the market. Instead of trying to build enough functionality to make it intuitive for anyone to use, I've decided to use what I built to read books in a more enjoyable manner on the weekends.

While the core functionality of the application is present, it requires some instruction and has a few gotchas. I would like to provide some context on the technology used and the complexity of the engineering work involved.

Resumed

I picked this project back up after reading some informative books. I've implemented speech highlighting and synthesis using a custom controler. There are still a few tricky bugs I'm working through, but it works very smoothly for many pdf file types.

If you're on mobile there is a bug where it doesn't save the pdf despite doing so during local testing.

No work has been done on many the controller intutive/nice to use.

Check out the latest version here

Info below is outdated

Demo:

Opening and listneing to a book

Adding a new book to the library

The easy part

The easiest part of this application is evident when visiting the website. Namely, the functionality of uploading a PDF and saving it to one's library (specific to the browser being used) with a neat 3D book cover, made possible in part through the open-source project: 3D Book Cover . The most challenging aspect was figuring out how to convert the first page of an uploaded PDF file into an image.

export async function getCoverImage(page: PDFPageProxy): Promise<string> {
  var scale = 1.5
  var viewport = page.getViewport({ scale: scale })

const canvas = document.createElement("canvas")
const context = canvas.getContext("2d")
if (!context) throw new Error("Error generating canvas context")

canvas.width = viewport.width
canvas.height = viewport.height

const renderContext = {
canvasContext: context,
viewport: viewport,
}

return page
.render(renderContext)
.promise.then(() => {
const dataURL = canvas.toDataURL("image/png")
return dataURL
})
.catch((error) => {
throw error
})
}

The harder part

Getting the PDF to display in a resizable manner with features relevant to reading was significantly more challenging. I delved deeply into the react-pdf source code to understand how to use their provided callback functions effectively. This allowed me to implement features such as saving the cover of a PDF as an image, tracking the most recent page the user has visited, and providing a user-friendly table of contents popover that appears when hovering over the eyeglass icon, which I designed using Figma. Additionally, I utilized React Window to render only a limited number of PDF pages at a time, resulting in an almost instantaneous load experience for the user.

The hardest part is most unoticable

Unfortunately, at this point in time, the application requires Microsoft's Edge browser for the read-aloud feature, which highlights the current word and sentence. Users must be familiar with Edge's Read Aloud feature. Additionally, after implementing React Window for faster load times, some of Edge's read-aloud features were disrupted. Specifically, users must double-tap on a word to activate the highlighting functionality properly. While using the assistive technology is currently unintuitive, when used correctly, it provides an assistive reading experience that, in my opinion, surpasses what's available on the market, including Speechify and Natural Reader.

Arguably, the most difficult aspect of this project wasn't the engineering work itself —though that was both challenging and an excellent learning experience— but rather figuring out how to highlight text on a PDF as flawlessly as the top software available. With very little information available online, it took me about a month of trial and error to find the right solution.

This process of experimenting with different approaches demonstrates a high level of perseverance in tackling a technical challenge.