I think a book has to be non-fiction if it’s going to be sold as non-fiction. I know: it’s a profound thought. If you’re writing a book about your life you need to recall all the conversations and events to the best of your ability. You should not make up that you’re multiracial or that you were adopted, if you’re white and still living in your biological parents’ house. That’s called realistic fiction, not a memoir. Certain obvious things, like that, we shouldn’t even have to ponder. There are some grey areas (like conversations), but I think if we try to get them as close to the truth as possible, it counts as a non-fiction memoir. Oh and saying you were in a gang, when in reality you only saw a movie about a gang is not as close to the truth as possible.
I do think we need genres as a reader or we would quickly get confused as to what’s real and what’s not. Our world is confusing enough, thank you. It helps you pick out the book you’re looking for. For instance, if you need to write a paper for forensics and you say some people died because of Voldemort, because the Harry Potter series was under non-fiction at the bookstore, you probably won’t get a good grade on that. It is important distinguish what is what. It helps the reader more than it hurts the author. Genres are necessary. There are too many books to know what each is about, or if you might be interested in reading it.